This post was originally published on this site

Many BigQuery customers rely on policy tags for protecting their sensitive information in BigQuery. Policy tags were the go-to solution for applying column-level access controls, allowing only users with the right permission to view sensitive columns like personally identifiable information (PII). It was a robust and effective system — for its time.

However, data ecosystems have grown in complexity, and the tools we use to help secure them need to evolve with them. New challenges include creating and managing a taxonomy that supports multiple tags across multiple regions and locations, enabling disaster recovery, and integrating with a broad centralized governance strategy.

To help you meet the needs of today’s data ecosystems, we’re excited to introduce the preview of data governance tags in BigQuery. Built on Google Cloud’s Identity and Access Manager’s (IAM) Resource Manager infrastructure, data governance tags provide a scalable, and robust method to help you manage access controls and protect your BigQuery column data. 

What are IAM data governance tags?

Data governance tags are a special type of Resource Manager tags.  You can create it by setting the purpose field to DATA_GOVERNANCE when creating a tag key in IAM, you designate it for use in BigQuery column-level security. You can create a hierarchical tree of data governance tags specifically for column-data governance purposes and apply them directly to your BigQuery columns. 

Why use data governance tags for column-level security?

  • Global scope, regional enforcement: Unlike policy tags (which are regional-only), data governance tags are global. You can define a single tag key:value pair (like “data_sensitivity:high”) at the organization level and use it across any project or region in your organization.

  • Managed disaster recovery: Security policies should persist during a failover. Data governance tags and their associated data policies are automatically replicated to secondary regions. If you need to switch regions, your security posture moves with you automatically.

  • Hierarchical security: You can now build a tree of tags up to five levels deep. This allows for inheritance and more granular classification (such as PII > Financial > CreditCardNumber).

  • Decoupled governance: You can tag your data to organize and classify it before you decide to enforce security. Access control only kicks in once you define a data policy for that tag, giving your team more flexibility during data onboarding.

Three steps to column-level security

Step 1: Create the tag key and values

1. Create data governance tag key: First you create an IAM tag key in Console, gcloud CLI, or API. The magic happens when you specify the purpose field as –purpose=DATA_GOVERNANCE for the tag key. This key change tells Google Cloud that this tag will be used for column-level security in BigQuery.

code_block
<ListValue: [StructValue([('code', '# Example: Creating a Data Governance tag key named "data_class"rngcloud resource-manager tags keys create data_class \rn –parent=projects/my-governance-project \rn –purpose=DATA_GOVERNANCE'), ('language', ''), ('caption', )])]>

2. Create tag values:

Once your data governance tag key has been created, you need to create specific tag values under the key that you will use to categorize/classify your column data.  One of the useful features of data governance tags is the ability to build a hierarchical tree of tag values. The tag-values tree allows you to create broad categories and then drill down into specific categories based on data type. You can go up to five levels deep for granular access control.

code_block
<ListValue: [StructValue([('code', '# Level 1: Create a tag value called "pii"rngcloud resource-manager tags values create pii \rn –parent=my-governance-project/data_classrnrnrn# Level 2: Create a child value under "pii" for "private" datarngcloud resource-manager tags values create private \rn –parent=my-governance-project/data_class/piirnrnrn# Level 3: Create another child tag value for "email" under "private"rn# You can go up to 5 levels deep for granular controlrngcloud resource-manager tags values create email \rn –parent=my-governance-project/data_class/private'), ('language', ''), ('caption', )])]>

Step 2: Attach tags to your columns via JSON schema

1. Export your existing schema

For existing tables, the most efficient way to manage tags is by updating the table schema using a JSON file and using API or BQ CLI because it allows you to tag multiple columns at once.

code_block
schema.json’), (‘language’, ”), (‘caption’, )])]>

2. Add the dataGovernanceTags to your JSON file

Open schema.json and add the tag mapping to your sensitive columns. Note the use of the namespaced key and the short name for the value.

code_block
<ListValue: [StructValue([('code', '[rn {rn "name": "user_email",rn "type": "STRING",rn "dataGovernanceTagsInfo": {rn "dataGovernanceTags": {rn "my-governance-project/data_class": "email" rn }rn }rn },rn {rn "name": "phone_number",rn "type": "STRING",rn "dataGovernanceTagsInfo": {rn "dataGovernanceTags": {rn "my-governance-project/data_class": "private"rn }rn }rn },rn {rn "name": "government_id",rn "type": "STRING",rn "dataGovernanceTagsInfo": {rn "dataGovernanceTags": {rn "my-governance-project/data_class": "pii"rn }rn }rn }rn]'), ('language', ''), ('caption', )])]>

3. Update the table:

Apply the schema to your BigQuery table.

code_block
<ListValue: [StructValue([('code', '# Overwrite the table schema with your newly tagged JSON file.rnbq update –project_id=my-data-project –schema=schema.json my_dataset.my_table'), ('language', ''), ('caption', )])]>

Alternatively you can also use SQL to bind data governance tags to BigQuery table columns.

code_block
<ListValue: [StructValue([('code', "CREATE OR REPLACE TABLE my_dataset.my_table(rn user_email STRINGrn OPTIONS (rn data_governance_tags = [('my-governance-project/data_class', 'email')]),rn );rnALTER TABLE my_dataset.my_tablernALTER COLUMN phone_numberrn SET OPTIONS (rn data_governance_tags = [('my-governance-project/data_class', 'private')]);rnALTER TABLE my_dataset.my_tablernADD COLUMN government_idrn STRINGrn OPTIONS (rn data_governance_tags = [('my-governance-project/data_class', 'pii')]);"), ('language', ''), ('caption', )])]>

You can also remove a column tag by setting it to [], for example:

code_block
<ListValue: [StructValue([('code', 'ALTER TABLE my_dataset.my_tablernALTER COLUMN phone_numberrn SET OPTIONS (rn data_governance_tags = []rn);'), ('language', ''), ('caption', )])]>

You can use information_schema COLUMNS view to see the columns tags:

code_block
<ListValue: [StructValue([('code', "SELECTrn column_name,rn data_governance_tags[SAFE_OFFSET(0)].key AS tag_key,rn data_governance_tags[SAFE_OFFSET(0)].value AS tag_value,rnFROM `my_project.my_dataset.INFORMATION_SCHEMA.COLUMNS`rnWHERE table_name = 'my_table'"), ('language', ''), ('caption', )])]>

The result is similar to the following:

code_block
<ListValue: [StructValue([('code', '+—————+———————————-+———–+rn| column_name | tag_key | tag_value |rn+—————+———————————-+———–+rn| user_email | my-governance-project/data_class | email |rn| phone_number | my-governance-project/data_class | private |rn| government_id | NULL | NULL |rn+—————+———————————-+———–+'), ('language', ''), ('caption', )])]>

Step 3: Create data policies

Finally, define a BigQuery data policy to govern access to these tagged columns. These policies explicitly reference the tag values you attached previously. Note that, while data governance tags are global, data policies are regional. 

To protect your data, the policy must be created in the same region where your BigQuery table is located. Once the policy is defined, access is only granted to the specified grantees; all others will be denied access to the sensitive column data.

Also, keep in mind that security in BigQuery is layered. For a data policy to be effective, the users (grantees) must first possess base-level access to the table itself (typically via a role like roles/bigquery.dataViewer). Data policy then acts as a second security layer, determining whether they view the raw, sensitive column data or a masked, obfuscated version.

Masking policy for ‘pii’ tagged column-data (SHA256 Masking):

code_block
<ListValue: [StructValue([('code', 'curl –request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \rn –header "Authorization: Bearer $(gcloud auth print-access-token)" \rn –header 'Accept: application/json' \rn –header 'Content-Type: application/json' \rn –data '{rn "dataPolicy": {rn "dataPolicyType": "DATA_MASKING_POLICY",rn "dataMaskingPolicy": { "predefinedExpression": "SHA256" },rn "grantees": [ "principalSet://goog/group/grp-sales@corp.com" ],rn "dataGovernanceTag": { "key": "myProject/data_class", "value": "pii" }rn },rn "dataPolicyId": "masking_policy_for_data_class_pii"rn}' \rn –compressed'), ('language', ''), ('caption', )])]>

 Raw access policy for ‘pii’ tagged column-data

code_block
<ListValue: [StructValue([('code', 'curl –request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \rn –header "Authorization: Bearer $(gcloud auth print-access-token)" \rn –header 'Accept: application/json' \rn –header 'Content-Type: application/json' \rn –data '{rn "dataPolicy": {rn "dataPolicyType": "RAW_DATA_ACCESS_POLICY",rn "grantees": [ "principal://goog/subject/abc@xyz.com" ],rn "dataGovernanceTag": { "key": "myProject/data_class", "value": "pii" }rn },rn "dataPolicyId": "raw_access_policy_data_class_pii"rn}' \rn –compressed'), ('language', ''), ('caption', )])]>

Masking policy for “private” tagged column data (NULL Masking):

code_block
<ListValue: [StructValue([('code', 'curl –request POST "https://bigquerydatapolicy.googleapis.com/v2/projects/myProject/locations/us-east1/dataPolicies" \rn –header "Authorization: Bearer $(gcloud auth print-access-token)" \rn –header 'Accept: application/json' \rn –header 'Content-Type: application/json' \rn –data '{rn "dataPolicy": {rn "dataPolicyType": "DATA_MASKING_POLICY",rn "dataMaskingPolicy": { "predefinedExpression": "ALWAYS_NULL" },rn "grantees": [ "principal://goog/subject/abc@xyz.com" ],rn "dataGovernanceTag": { "key": "myProject/data_class", "value": "private" }rn },rn "dataPolicyId": "null_policy_data_class_private"rn}' \rn –compressed'), ('language', ''), ('caption', )])]>

With these three steps, your column data is now protected. The next time a principal queries your BigQuery table, our authorization engine automatically evaluates their identity against your data policies. If the principal is part of the policy, they get to see the masked or raw data as per the policy; if they are not, then they will be denied access. 

Get started today

Data governance tags are a powerful new tool to enhance your data security and governance strategy in BigQuery. We are continuously working to enhance data governance capabilities in BigQuery. Future updates include support for using SQL to create tags and tag based policies, ability to attach multiple tags to a single column,  ability to define policies based on combinations of tags, and deeper integrations with services like Knowledge Catalog.

You can start tagging your columns and defining fine-grained access controls at scale. To learn more, dive into the Data Governance Tags documentation.