Every data team knows the moment. Someone opens a table, sees a column called cust_seg_flg, and has to go ask around to find out what it means, whether it’s safe to use, and whether anyone has already answered that question in another dashboard three teams over. Multiply that by thousands of tables and views, and you get the real cost of governance debt: not a compliance failure, but a daily tax on every person trying to do honest work with your data.
Most governance tooling today is reactive. You scan for problems, you get a report, someone opens a ticket, and three weeks later a column gets a description. The Governance Agent project (built on Google Cloud Knowledge Catalog, BigQuery, and column-level lineage) takes a different starting point: if a table upstream is already documented, tagged, and trusted, why should every downstream view have to earn that trust from scratch, by hand, every time?
This post is about that shift, from governance as an audit you dread to governance that keeps itself current in the background.
The problem in plain terms
Data estates grow through pipelines. Raw tables get joined, filtered, and reshaped into views, and those views feed more views. Somewhere in that chain, the original context (what a column means, whether it’s PII, what quality bar it’s held to) tends to get lost. It just doesn’t travel.
The result is a familiar pattern: a handful of gold tables are well governed because someone invested real time in them, and everything built downstream of them is progressively less documented, less tagged, and less trustworthy, even when the underlying data hasn’t actually gotten worse. The governance quality of a table ends up depending on how long ago someone cared about it, not on how the data is actually being used today.
As data flows through a company, it gets combined, filtered, and reshaped for different teams to use. But somewhere along that journey, the important context—like what a piece of information means, whether it contains private details, or if it’s accurate—gets left behind. The metadata simply doesn’t travel with the data through the progression of data assets within the ecosystem.
The result is a familiar pattern: a company will have a few perfectly documented “core” datasets because someone invested time in them, but everything built on top of them becomes a mystery. The data itself hasn’t gone bad, but without the original context, people stop trusting it. Ultimately, data is only considered reliable if someone manually updated its metadata recently, rather than because of what it actually contains.
What the agent actually does
The core idea is straightforward: use column-level lineage to figure out where a column came from, and propagate the governance metadata that already exists upstream, rather than asking a human to re-derive it.
Concretely, it handles four things:
Descriptions. If transactions.customer_id has a clear description upstream, and a downstream view pulls that column through two or three hops of joins, the agent traces that lineage and proposes the same description downstream. When a column isn’t a straight passthrough (it’s a SUM(), a CASE WHEN, a COALESCE), the agent reads the actual SQL that generated it and writes a description that reflects the transformation, instead of copying an upstream description that no longer applies.
Business glossary terms. Technical column names rarely match the business language people actually use. The agent uses semantic similarity to map columns to a controlled glossary, and can also read unstructured documents (a PDF policy, a product spec, a markdown design doc) to find explicit definitions rather than guessing from column names alone.
Policy tags. This is the one that matters most for risk. If a column is tagged as PII upstream, the agent traces where that data flows and recommends the same tag downstream, along with a summary of who currently has read access and what masking rules apply. It also checks whether a transformation looks like a “straight pull” (the sensitive value passed through unchanged) versus something that’s been aggregated or anonymized, so it isn’t blindly stamping PII tags on data that no longer carries the risk.
Trust and data quality scores. Rather than treating every view as an unknown, the agent derives a trust score based on the Data Quality and Profiling results of its upstream sources, and gives credit when it detects that a transformation actually improved data quality (deduplication, null handling, and so on).
Every one of these runs through a confidence threshold before anything gets applied. The system is explicit about not inferring PII status or glossary mappings without solid grounding. If the evidence is weak, the propagation doesn’t happen automatically. That’s a deliberate design choice: the agent is meant to close obvious gaps quickly, not make judgment calls that a person should be making.
Why proactive is the right word, and not a stretch
Proactive governance doesn’t mean predicting the future. It means the governance work happens as data moves, instead of waiting for a scheduled review or a compliance incident to trigger it. In practice, that shows up in three ways:
-
New views inherit context automatically, instead of starting undocumented and waiting for someone to notice.
-
Sensitive data is flagged as it flows, not discovered after it’s already been queried by twelve people who didn’t know they needed a masking policy.
-
Stewards spend their time on judgment calls, like ambiguous mappings or new glossary terms, instead of repetitive column-by-column tagging that a lineage graph could have told you.
None of this replaces a data steward. It changes what a data steward’s day looks like: fewer hours spent typing descriptions into a UI, more hours spent deciding what should count as a business term or whether an edge case needs a policy exception.
When lineage runs out, bring your own context
Lineage is powerful, but it isn’t complete. Plenty of tables have no clean upstream source to inherit from: a newly ingested dataset, a one-off import, a table that predates whatever lineage tracking you have in place. For those, the agent doesn’t just shrug. It lets you point it at your own documents (a PDF policy, a product spec, a markdown design doc, even a spreadsheet or a screenshot of a data dictionary) and uses that as grounding instead.
There are three ways to feed it context, and which one you pick depends on the size of what you’re handing over. For a short document, you can inject the full text directly into the prompt. For something long, like a fifty-page data classification policy, the agent chunks it, embeds it, and retrieves only the passage relevant to the specific column it’s describing, so you’re not paying to re-read the whole document for every field. And if your organization already has a proper document repository indexed in Vertex AI Search, the agent can query that directly instead of re-processing files every time.
The part worth calling out is how conservative the grounding is. This isn’t “read the doc and take a guess.” The instructions given to the model are explicit: if a column isn’t clearly defined in the document you provided, it has to say so and stop, not fill in the gap with a plausible-sounding guess. That rule is strict for policy tags and glossary terms in particular. A column only gets marked as PII if the document says so in plain language, like an explicit “PII: Y” flag or a named sensitivity section. No inference from column names, no “this sounds like it might be sensitive.”
That distinction matters more than it sounds like it should. A tool that infers PII status when it’s uncertain is a tool that will eventually mask a column that didn’t need it, or worse, wave through one that did. Making “I don’t know” a valid answer is what makes the automation trustworthy enough to run without someone re-checking every single output.
Two signals, not one: lineage plus insights
Lineage is the primary signal, but it isn’t the only one the agent listens to, and it’s worth being precise about why. The Data Lineage API only knows what a job explicitly recorded. If a table was built through a well-instrumented pipeline, that’s a clean, high-confidence trail. But plenty of real estates have gaps: a table that predates good lineage capture, a transformation that ran outside the tracked jobs, a relationship that technically exists but was never logged as such.
For those gaps, the agent has a second pass. It can trigger a Knowledge Catalog Data Documentation scan (an AI-driven analysis Gemini runs over a table or dataset) which infers relationships and column meaning even without a clean SQL trail behind them. Those inferred relationships get extracted and cached locally, then loaded into the same traversal engine that handles lineage, so both signals are checked together rather than living in separate systems a steward has to reconcile by hand.
The order matters. Standard lineage runs first, since it’s grounded in an actual recorded job. The Insights pass runs second, filling in only what lineage didn’t find, and anything it contributes is explicitly tagged as coming from that source rather than blended in silently. If you’re auditing a propagated description or tag later, you can tell whether it came from a hard lineage link or an inferred one. There’s a dedicated end-to-end flow for this path (trigger the scan, wait for it, extract the results, apply them) so it isn’t a manual side-quest bolted onto the main workflow.
The practical effect: an incomplete or newly onboarded pipeline still gets useful propagation on day one, instead of waiting until lineage coverage catches up.
What it looks like day to day
The project ships with both a Gradio-based dashboard and a CLI, which matters more than it sounds like it should. A steward reviewing a handful of tables before a demo will want the dashboard: run a scan, see which tables have metadata gaps, preview a proposed description or tag, and approve it with a click. A platform team that wants this running as part of a nightly job or a CI/CD pipeline will use the CLI, scripting steward_cli scan, apply, and policy-propagate commands the same way they’d script any other pipeline step.
That dual interface reflects a real operational choice: governance tooling that only works from a UI never gets automated, and governance tooling that only works from a CLI never gets adopted by the people closest to the data.
Where this needs a human in the loop
Worth saying plainly: this is not a “set it and forget it” system, and it shouldn’t be treated as one. Lineage confidence scoring can still get things wrong, especially across renamed columns or unusual joins. Semantic mismatch checks catch obvious errors (a date column shouldn’t inherit a description from an id column) but they’re heuristics, not guarantees. Every propagation is designed to be previewed before it’s applied, and that preview step isn’t a formality, it’s the actual safety mechanism.
The honest pitch here isn’t “governance without effort.” It’s “governance where the effort goes to the right five percent of decisions instead of the repetitive ninety-five percent.”
Customer Testimonial
“As custodians of the VodafoneThree UK Datahub, one of our biggest challenges is that a significant proportion of our data estate remains undocumented or inconsistently labelled. This creates friction for data discovery, slows down delivery teams, and limits the value we can unlock from AI solutions built on top of our data. The Data Steward Agent changes that. By combining cataloguing, lineage, and automated metadata propagation, it enables us to focus governance effort where it adds the most value while automatically carrying trusted context downstream. Rather than manually reviewing thousands of tables, we can concentrate on governing source datasets and allow lineage to scale that knowledge across the platform. We estimate this approach can reduce cataloguing effort by up to 75%, while significantly improving data discoverability, trust, and AI readiness across the UK Datahub.” – Radina-Paola Ivanova, GenAI Engineer, VodafoneThree UK Datahub
The takeaway
Governance debt compounds the same way technical debt does: quietly, until someone downstream hits it at the worst possible time. The value of an approach like this isn’t that it makes governance disappear as a concern. It’s that it moves the work upstream, literally, so that context and controls travel with the data instead of being reconstructed from scratch every time someone builds a new view.
For teams sitting on years of undocumented BigQuery estates, that’s not a nice-to-have. It’s the difference between governance being something you catch up on twice a year, and governance being something that just keeps pace with how fast your data actually moves.