Zero-shot predictions in Pandas using the new BigQuery Accessor

Imagine running advanced machine learning workflows—from forecasting and classification to high-dimensional embedding generation—without ever leaving your familiar pandas DataFrame. With the new .bigquery accessor from BigQuery DataFrames (BigFrames), that vision is now a reality. You can seamlessly invoke BigQuery’s enterprise-grade AI functions directly within pandas, scaling your data science and ML pipelines on cloud infrastructure .

Take linear regression as an example. Traditionally, a pandas user who needs to run regression would have to extract data locally, manually select features, and train a custom scikit-learn model like this:

Now, with the .bigquery accessor, you can bypass the local training process entirely and leverage powerful ai.predict() directly on your data in BigQuery:

As you can see, the ai.predict() is able to perform regression tasks directly over two pandas dataframes with a single function call, which greatly simplifies the entire ML process.

How the BigQuery accessor works

Under the hood, the .bigquery accessor uses BigQuery DataFrames library to transparently push your computations to Google Cloud. When you call an AI method like .ai.predict(), it doesn’t compute the regression locally. Instead, it translates the pandas operation into Google Standard SQL, executes the corresponding foundational model (like TabFM) natively on BigQuery’s infrastructure, and seamlessly returns the result as a standard pandas DataFrame. This allows you to easily process local datasets with powerful offerings unique to BigQuery.

The following sections detail additional executable code examples. You can also find them all in this notebook.

Environment setup

Before executing the code samples in this post, ensure you have a Google Cloud project with the following roles granted (see more at permissions for AI functions):

• Run query jobs: BigQuery Job User (roles/bigquery.jobUser)

• Access a foundation model in Gemini Enterprise Agent Platform: Agent Platform User (roles/aiplatform.user)

Next, install pandas and bigframes in your Jupyter environment. To enable the AI accessors, simply import the bigframes library and specify your Google Cloud project ID. This ensures all subsequent AI function calls are routed through your authorized project:

Once configured, you can call BigQuery AI functions directly on standard pandas objects.

Functions on pandas series

AI functions attached to a pandas `Series` accept single-column input sequences.

Example: AI.EMBED

Generating vector embeddings is a fundamental step in many modern NLP and semantic search pipelines. By using the `ai.embed()` function, you can compute embeddings for an entire column of text in a single, clean expression.

Behind the scenes, BigFrames converts the pandas Series for processing in BigQuery and casts the output back into a native pandas Series. From the developer’s perspective, the workflow remains entirely within pandas.

Notice that the output of this operation is a native pandas `Series` containing the resulting embedding list. BigFrames handles the temporary BigQuery conversions, model execution, and final data formatting, ensuring that you never have to leave your familiar pandas interface.

Example: AI.SIMILARITY

When you need to calculate semantic closeness, the `ai.similarity()` function computes cosine similarity scores between your series elements and a target. This target can be either a scalar literal string or another series. Here is how you can compare a pandas `Series` against a scalar string to find matching concepts:

Similarly, you can perform pairwise comparisons between two distinct pandas `Series` objects to find semantic correlations across different datasets:

In both cases, the returned object is a standard pandas `Series` with floating-point similarity scores, making it immediately compatible with downstream pandas methods like filtering or sorting.

Functions on pandas DataFrames

DataFrame AI functions operate on multi-column structured tables or handle complex prompts involving multiple data inputs.

Example: AI.FORECAST

Time-series forecasting typically requires a lot of boilerplate code to format timestamps, group entities, and train a model. The `ai.forecast()` function performs time-series forecasting across grouped structured data with simple, high-level parameters. To demonstrate this, you first prepare the input sample DataFrame representing historical sales data:

With your data prepared, you can call `ai.forecast()` directly on the DataFrame. Here, you specify the target metric, timestamp column, grouping key, and a prediction horizon of three days:

The returned DataFrame provides complete forecast values alongside statistical parameters such as confidence levels and prediction intervals. Internally, this triggers a TimesFM model that performs the trend analysis, returning neatly structured results back to your local environment as a pandas DataFrame. You can also choose the model version with the model argument. For example, pass model=”TimesFM 3.0″ to use the newest TimesFM model, now in Preview in BigQuery, which ranks #1 on the GIFT-Eval, FEV-Bench, and TIME forecasting benchmarks.

Example: AI.IF

Filtering rows on a natural-language condition is one of the most common generative AI tasks. The `ai.if_()` function evaluates a prompt built from literal strings and DataFrame columns, and returns True or False for each row:

Because `ai.if_()` returns a boolean Series, you can use it directly as a pandas filter mask, for example `df[df.bigquery.ai.if_(prompt)]`. `ai.if_()` runs in optimized mode by default (optimization_mode=”minimize_cost”): a local, distilled model processes the majority of rows, and only the rest are sent to Gemini, which reduces cost and latency on large datasets. To send every row to the LLM instead, set `optimization_mode` to `”maximize_quality”`.

Get started

The AI functions provided by bigquery accessor unlock advanced machine learning tools directly inside your native data structures, enabling you to solve complex AI challenges without leaving the Python ecosystem you already know.

Currently supported AI functions include the following (see the documentation for full details):

Pandas Series

• embed(): Generates vector embeddings to a single column.

• generate_embedding(): Generates vector embeddings to a table.

• generate_table(): Performs natural language tasks and generates structured outputs.

• generate_text(): Performs natural language tasks and generates free-form texts.

• similarity(): Computes semantic closeness (cosine similarity) between entries.

Pandas DataFrame

• classify(): Performs zero-shot classification.

• forecast(): Runs time-series forecasting across grouped data using TimesFM, including the new TimesFM 3.0.

• generate(): Evaluates dynamic prompts across multiple columns to generate text responses.

• generate_bool(): Evaluates natural language conditions and returns booleans (True/False).

• generate_double(): Evaluates a prompt and returns floating-point numbers.

• generate_int(): Evaluates a prompt and returns integers.

• if_(): Filters rows on a natural-language condition, using optimized mode by default to reduce cost and latency.

• predict(): Runs predictive modeling, such as regression, using TabFM.

• score(): Rates inputs based on a scoring system that you described.

Whether you want to generate embeddings or run full time-series forecasts, BigQuery AI functions inside pandas make scaling machine learning effortless. Give it a spin and let your data do the heavy lifting!