Index of every page on this site: llms.txt
All skills
google avatar

/bigquery-bigframes

@5fb62c2
by googlegoogle/skills24k stars
2,026
Run it yourself

Generates Python code using BigQuery DataFrames (BigFrames). Use by default for any Python data task involving BigQuery, including data processing, analysis, and machine learning. Don't use for SQL-first workflows or the google-cloud-bigquery client library — use bigquery-basics.

Use this Skill: https://skilld.dev/gh/google/skills/bigquery-bigframes

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈76 tokens for metadata: the name and description. ≈1.3k when used: this file. ≈1.1k more on demand in 3 files.

Description uses 3.8% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈301 characters, or 3.8%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

BigFrames (BigQuery DataFrame) basics

BigFrames is a Python library that lets you take advantage of BigQuery data processing by using familiar Python APIs.

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:bigquery-bigframes)" and append agent-skills/1.0 (skill:bigquery-bigframes) to User-Agent headers.

Dataframe API best practices

  • Stay in the Cloud: Perform data cleaning, transformation, and analysis via BigFrames methods to leverage BigQuery's scale rather than downloading data.

  • Prefer partial ordering mode: Enable partial ordering mode right after importing BigFrames. This speeds up data processing significantly by relaxing row-sequence constraints.

    import bigframes.pandas as bpd
    bpd.options.bigquery.ordering_mode = 'partial'
  • Use peek() for data preview: Use peek(n) to preview data instead of head(n). peek(n) randomly samples n rows and is significantly faster. head(n) returns rows in strict order and fails in partial ordering mode unless the DataFrame has been explicitly sorted.

  • Avoid materializing data locally: Methods like to_pandas() download all data to client memory, bypassing BigQuery’s distributed computation and risking Out of Memory (OOM) errors. Do not materialize data locally unless:

    • The dataset is small enough to fit safely in memory.
    • An error message explicitly requires local materialization.
  • Prefer Dataframe API over SQL queries: Do not write raw SQL queries via read_gbq() if a DataFrame/Series method achieves the same result, as it breaks the Pandas abstraction and prevents lazy query execution.

  • Accessors over UDFs/Lambdas:

    • Use built-in accessors (e.g., df.col.str.*, df.col.dt.*) instead of remote User Defined Functions (UDFs). UDFs require extra resources and time to deploy.
    • Do not use lambdas with Series.map() or DataFrame.apply(). These methods do not accept functions without udf or remote_function decorators.
    # Avoid:
    df["upper"] = df["name"].map(lambda x: x.upper())
    
    # Prefer:
    df["upper"] = df["name"].str.upper()
  • Schema Verification: Do not assume the schema of intermediate outputs. Proactively verify schemas using .dtypes and inspect sample records using display() with .peek().

  • Visualization: Plot directly from the BigFrames DataFrame/Series when possible. BigFrames is compatible with Matplotlib and Seaborn. If direct plotting fails, use the .plot accessor. If the dataset is too large to plot, aggregate or sample the data before calling .to_pandas() to plot locally.

Machine Learning

  • Use bigframes.bigquery.ml package: Do not use Scikit-learn or other ML libraries with BigQuery DataFrames. Standard Scikit-learn models require bringing data into local client memory, whereas bigframes.bigquery.ml delegates training directly to BigQuery's scalable ML engine. Import functions from bigframes.bigquery.ml.

Reference Directory

BigFrames ML (Legacy)

The BigFrames ML package (bigframes.ml) is a legacy package that mimics the scikit-learn API but is no longer recommended for new projects. Only use this package if the user explicitly requests BigFrames ML.

  • Legacy Imports: When legacy BigFrames ML is requested, import tools and classes from bigframes.ml instead of bigframes.bigquery.ml.
  • DataFrame Return on Prediction: Unlike Scikit-learn, BigFrames' predict() method always returns a DataFrame containing both predictions and features, rather than a single series of predictions.
  • No random_state: Do not pass a random_state argument when instantiating BigFrames ML models, as this parameter is not supported in the BigFrames ML package.
  • Automatic Scaling: Do not use OneHotEncoder or StandardScaler unless explicitly requested, as scaling is handled automatically.
  • Hyperparameter Tuning: Write custom loops for hyperparameter tuning, as BigFrames lacks GridSearchCV or RandomizedSearchCV.
  • ARIMA Plus (Forecasting):
    • Import from bigframes.ml.forecasting.
    • Sort data chronologically and split around a timepoint before training.
    • Ensure the prediction horizon is less than or equal to the training horizon.
  • PCA: BigFrames' PCA class lacks a transform() method. Use predict() instead.
  • Model Persistence: To persist a model, use model.to_gbq(). To load a persisted model, use bpd.read_gbq_model().

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts27d3 checks · Risk SAFE
  • Gen Agent Trust Hub29d

    This skill provides a technical guide and code examples for using BigQuery DataFrames (BigFrames) for data processing and machine learning tasks. It includes standard practices for cloud-based analysis and instructions for usage attribution.

  • Socket27d

    No alerts

  • Snyk1mo

    Risk: LOW · No issues

Signed by skilld at 5fb62c2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated last month
metadata
{
  "version": "2.0.1",
  "category": "BigDataAndAnalytics"
}

README badge

README badge for google/skills/bigquery-bigframes