This post was originally published on this site

For data science, data engineering, and machine learning teams, unlocking the value of unstructured data, such as images, videos, audio, and documents, has historically meant managing siloed data pipelines for structured and unstructured data, and for analytics and AI/ML use cases. Even when files remain in Cloud Storage, running inference requires teams to orchestrate external machine learning workloads, only to face the complex, brittle challenge of stitching those structured inference outputs back into the data warehouse for analysis. This post-hoc data-stitching introduces significant operational overhead and database misalignment.

ObjectRef and multimodal data analytics in BigQuery are now in General Availability (GA), establishing BigQuery as a comprehensive, multimodal agentic data platform.

With ObjectRef, BigQuery natively bridges the gap between Cloud Storage and your analytical engine. You can represent unstructured data as native ObjectRef columns directly alongside your traditional relational columns. This unlocks a massive advantage over standalone model APIs by enabling you to pass multimodal, multi-column inputs directly to Gemini models in a single SQL query or natural language query using conversational analytics. ObjectRef integrations with BigQuery AI, conversational analytics in BigQuery and Gemini Enterprise enable you to easily combine Cloud Storage files with rich, structured database context, metadata, timestamps and custom agent instructions.

You can now leverage BigQuery’s core strengths (fully serverless compute, low total cost of ownership, and built-in at-scale inference) to build, run, and govern powerful unstructured-to-structured, unstructured-to-unstructured and fully multimodal-to-multimodal pipelines without separate compute clusters. Crucially, the extracted features and annotations can be combined directly with the unstructured object itself by storing the ObjectRef in the same row. Using this combined data directly in conversational analytics and AI agents avoids the loss of nuanced data and context that typically occurs when customers transform unstructured files into purely structured formats. Customers are already using this approach to generate embeddings for multimodal vector search, perform at-scale classification using GenAI, and execute precise data and feature extraction from vast archives of documents and media, all via standard SQL.

code_block
<ListValue: [StructValue([('code', 'SELECTrnAI.AGG(image_ref, "Summarize the major categories of pictures")rnFROMrn`bigquery-public-data.cymbal_pets.products`;'), ('language', ''), ('caption', )])]>

ObjectRef brings structure to the unstructured

ObjectRef is a built-in, predefined STRUCT data type that points to objects outside of BigQuery.

code_block
<ListValue: [StructValue([('code', '– ObjectRef is a special STRUCT in BigQuery rnstruct {rn uri string,rn authorizer string,rn version string, rn details json { rntgcs_metadata jsonrn }rn }'), ('language', ''), ('caption', )])]>

By storing these pointers natively in standard BigQuery tables alongside traditional columns, BigQuery treats unstructured files as first-class data citizens. Furthermore, this architecture supports independent data lifecycles: your GCS data and BigQuery tables can change independently without affecting the underlying data in the other system.

This approach serves multiple data personas by offering powerful advantages:

  • Zero-copy architecture: Keep your high-volume media files securely in Cloud Storage and query them instantly in BigQuery, respecting data gravity and avoiding costly data duplication.
  • SQL and Python interoperability: You can query unstructured data using standard SQL or BigQuery DataFrames. Crucially, you can pass ObjectRefs into BigQuery Python UDFs, allowing you to bring open-source Python libraries, like OpenCV, directly to your data to execute custom logic without moving it.
  • Independent lifecycles and versioning: ObjectRefs ensure reproducible inference by tracking specific GCS generation numbers. If the underlying GCS file changes, your query can still reference the exact version of the file that existed when the row was written.
  • Unified governance: You can secure, audit, and share access to both structured columns and their related unstructured files using BigQuery’s existing security policies and IAM controls.

Create full-fidelity ETL with SQL

Since launching ObjectRef in preview, a powerful, highly effective pattern has emerged among our users: building full-fidelity ETL pipelines entirely with SQL.

image1

Teams are leveraging BigQuery’s serverless architecture to integrate AI/ML directly into their data environments, eliminating the need for separate compute clusters to handle data movement and inference. Managing unstructured data in GCS as standard BigQuery columns allows users to apply core data engineering best practices, such as building idempotent pipelines, ensuring reproducibility, and executing seamless backfills, directly to images, video, audio, and documents.

Once data is robustly prepared, teams use Gemini within BigQuery to execute full-fidelity ETL. They unlock dual-layer analytics by extracting features and annotating unstructured data while keeping the original ObjectRef anchored in the same row. The structured outputs power fast filtering, reporting and BI dashboards, while the attached unstructured objects remain available for deep, context-heavy reasoning. For example, you might extract sentiment scores and topic tags from an archive of customer call recordings to quickly flag at-risk accounts on a BI dashboard (the structured layer). Simultaneously, a conversational analytics agent evaluating escalation processes can seamlessly fall back to the original audio files to analyze the caller and customer service representative interactions, preventing the loss of critical nuance that typically occurs in unstructured-to-structured ETL pipelines.

Proven use cases from industry leaders

“At Snap, the safety of our community is a top priority, and we aim to limit users’ exposure to content that violates our Community Guidelines. We regularly review samples of public posts, and by leveraging ObjectRef with BigQuery AI, we transitioned from simple random sampling to post-stratification sampling. This transition improved the efficiency of our human review processes, enabling us to reinvest reclaimed capacity into new and expanded workflows that further enhance our ability to detect and address potentially harmful content.” – Carlos Ortiz, Data Scientist, Snap

During the preview phase, we saw adoption across a variety of critical workloads. Customers have successfully transitioned these experimental pipelines into production, transforming unstructured data into actionable, structured insights:

  • Content moderation: Online platforms leverage BigQuery to integrate multimedia assets with standard telemetry data in a single pipeline to fundamentally enhance their moderation and safety outcomes.

  • Proactive threat detection: Enterprise security teams use BigQuery to analyze visual and structural web components alongside network logs, driving critical advancements in proactive threat identification.

  • Inventory compliance: Online retailers extract text and context directly from vast image repositories to automatically flag regulatory or brand compliance issues at scale.

  • Customer experience: Service-oriented enterprises analyze audio interactions between support agents and customers with BigQuery AI to evaluate customer sentiment and improve service delivery.

New in GA

We streamlined the developer experience for the General Availability release, focusing on support for a wide variety of customer use cases:

  • Flexible authentication:

    • In GA, we introduced End User Credentials (EUC) support for ObjectRefs. Users can now quickly query and explore unstructured data using their personal IAM identity directly for ad-hoc analysis.

    • In preview, querying unstructured files required setting up explicit BigQuery connections for ObjectRefs. While these connections remain fully supported and are the recommended best practice for robust, service-account-driven production pipelines, we recognize they require additional setup.

  • Unified support for your Cloud Storage metadata: We know your unstructured data’s metadata lives in various places across Google Cloud. ObjectRef now natively supports the full spectrum of these sources. Whether your metadata currently resides in BigQuery object tables, Cloud Storage insights datasets, or simply as raw string URIs within standard BigQuery tables, you can seamlessly create and manage ObjectRefs to analyze the underlying data.

  • Array support for comparative analytics: We introduced native support for processing ObjectRefs contained within arrays. This enables entirely new classes of workloads, such as comparing multiple documents against one another, analyzing sequences of images, or batching items in a single semantic call.

See it in action: End-to-end multimodal analytics

To illustrate the power of these GA updates, let’s look at how you can build real-world, multimodal pipelines using our streamlined SQL operators.

1. Full-fidelity ETL: Call center audio analysis

Analyzing customer service calls historically required moving audio files to specialized transcription and natural language processing clusters. With ObjectRef, you can parse audio files, extract structured business intelligence and add them directly as standard relational columns.

Use the GA OBJ.MAKE_REF() function to automatically detect the audio format from a raw string URI and BigQuery’s managed AI functions for well known use-cases. For example, you can use AI.CLASSIFY() to categorize customer call types. By default, BigQuery uses Optimized Mode for AI Functions to reduce the number of tokens consumed.

code_block
<ListValue: [StructValue([('code', 'CREATE OR REPLACE TABLE call_center.annotated_call_logs ASrnSELECT rn customer_id,rn call_date,rn OBJ.MAKE_REF(audio_uri) as ref,rn AI.CLASSIFY(rn OBJ.MAKE_REF(audio_uri), [rn "Lost or stolen card", rn "Declined card payment", rn "Activate my card",rn "Order physical card",rn "Cancel transfer",rn "Password or pin forgotten",rn "Virtual card not working",rn "Card swallowed by ATM",rn "Balance not updated after bank transfer",rn "Balance not updated after check or cash deposit",rn "Extra charge on statement",rn "Transfer fee charged"rn ]rn ) AS customer_issue_type,rn FROM rn call_center.raw_call_logs;'), ('language', ''), ('caption', )])]>

This table can serve as a multimodal data source for a BigQuery conversational analytics agent. The agent translates natural language queries into SQL using BigQuery AI functions to analyze audio files when additional information is needed.

2. At-scale gen AI filtering: Product compliance

Data engineers frequently need to filter massive datasets based on the contents of images or documents. Instead of running heavy generative queries and parsing text outputs, you can now use the declarative AI.IF() operator natively alongside your standard SQL in WHERE clauses. The default Optimized Mode reduces the number of tokens consumed to further optimize your workloads.
code_block
= CURRENT_DATE() – 1 ANDrn AI.IF((“Does this product image contain a restricted item or policy violation?”, OBJ.MAKE_REF(image_uri))’), (‘language’, ”), (‘caption’, )])]>

3. Multi-object reasoning: Proactive threat detection

With native array support in GA, you can now pass an array of ObjectRefs into a single Gemini call to analyze multiple files simultaneously using Gemini.
This unlocks complex comparative logic directly in SQL. For example, a security team can pass a screenshot of their official login page and a screenshot of a newly discovered, suspicious webpage to Gemini simultaneously to detect phishing attempts at scale:
code_block
‘is_phishing BOOLEAN, visual_discrepancies STRING’rn ) AS assessmentrn FROM rn cybersecurity.network_ui_logsrn);’), (‘language’, ”), (‘caption’, )])]>

Start analyzing today

BigQuery ObjectRef is now generally available on all BigQuery editions and on-demand pricing tiers.

If you are currently managing complex, fragmented infrastructure just to perform machine learning inference on your Cloud Storage files, it’s time to consolidate. Bring AI directly to your storage. Start leveraging your existing data engineering practices to build robust, scalable, and secure multimodal analytics pipelines directly within BigQuery.

To get started with ObjectRef and multimodal analytics, check out these resources: