How to build a data governance pipeline with agents, from raw to certified tables

Quick answer

Put your governance rules in one policy file in the repository: which data counts as PII, how each class is masked, how long it is kept, and what a table needs before it is certified. A Nori agent reads the file, tags every column, generates masked views and tests, writes lineage docs, and opens a pull request for each change. A table is marked certified only when every check in the policy passes and a named owner approves.

Why the policy lives in a file

Most governance programs fail at the same point. The rules are written in a document, and the warehouse is changed by people who have not read it. A policy file closes the gap. It is versioned, reviewed like code, and precise enough for an agent and a test runner to apply the same way every time.

The agent does not decide what your policy is. Your data owner and whoever is responsible for privacy write the rules. The agent applies them to every table, shows its work, and stops where the file does not give an answer.

# governance/policy.yml
classifications:
  pii_direct:
    description: Identifies a person on its own.
    examples: [email, phone, full_name, street_address]
    masking: sha256_salted        # joinable, not readable
  pii_sensitive:
    description: Government or financial identifiers.
    examples: [ssn, tax_id, bank_account, date_of_birth]
    masking: redact               # replaced with null
  pii_indirect:
    description: Identifies a person when combined.
    examples: [ip_address, postal_code, device_id]
    masking: truncate             # /24 for IPs, first 3 chars for postal codes
  internal:
    masking: none

retention:
  default_days: 1095
  overrides:
    - match: {schema: silver, table: support_tickets}
      days: 730
    - match: {classification: pii_sensitive}
      days: 365

access:
  unmasked_roles: [PRIVACY_OFFICE, FINANCE_OPS]
  default_view: masked

certification:
  requires:
    - owner_assigned
    - every_column_classified
    - primary_key_tests_pass
    - freshness_within_sla
    - masked_view_exists_if_pii
    - retention_job_configured
    - lineage_doc_current
  approver: table_owner

How to build the pipeline, step by step

1. Write the policy file. Start from the example above and edit it with your privacy and data owners. Merge it through a normal reviewed pull request, with those owners as required reviewers in CODEOWNERS.
2. Give the agent a governance role. It needs read access to table metadata and sample values in the layers it tags, and write access only to a development schema. Add the credentials to Nori as an integration scoped to the data agent.
3. Ask for the first pass in Slack. @nori apply governance/policy.yml to everything in silver. Classify columns, generate masked views and tests, and open one PR per source system.
4. The agent classifies every column. It uses column names, data types and a sample of values. A column called contact that holds email addresses is tagged pii_direct because of what is in it, not what it is called. Each tag records the evidence.
5. It generates the artifacts. For each table it writes dbt meta tags, a masked view, tests that fail when an untagged column appears, a retention job entry, and a lineage doc from the dbt graph.
6. Owners review. Uncertain classifications are listed at the top of the pull request as questions. The table owner answers in the review or in the Slack thread, and the agent updates the branch.
7. Certify. When all seven requirements pass, the agent opens a small pull request that sets certified: true and requests the table owner as approver. Merging it is the certification.
8. Keep it true. A nightly automation rechecks every certified table. A webhook from your git host runs the same check on every pull request that touches a model.

Worked example: one table from raw to certified

Take silver.customers, built from a CRM export. The agent's classification, as it appears in the pull request description:

The notes row is the important one. The policy has no rule for free text, so the agent asks rather than picking. The owner replies in the thread: Treat free text as pii_direct and redact it in the masked view. Add that to the policy. The agent adds a free_text rule to policy.yml in the same pull request, so the privacy owner reviews the rule change too.

The generated tags go into the model's YAML, where dbt docs and your catalog can read them:

# models/silver/_customers.yml (generated section)
models:
  - name: customers
    meta:
      owner: "@maria.k"
      certified: false
      retention_days: 365        # pii_sensitive override applies
      policy_version: "2026-11-10"
    columns:
      - name: email
        meta: {classification: pii_direct, masking: sha256_salted}
      - name: contact
        meta: {classification: pii_direct, masking: sha256_salted}
      - name: date_of_birth
        meta: {classification: pii_sensitive, masking: redact}
      - name: postal_code
        meta: {classification: pii_indirect, masking: truncate}
      - name: notes
        meta: {classification: pii_direct, masking: redact}

And the masked view, which is what analysts query by default:

-- models/gold/customers_masked.sql (generated from governance/policy.yml)
select
    customer_id,
    sha2(concat(lower(trim(email)), '{{ var("pii_salt") }}'), 256)  as email_hash,
    sha2(concat(contact, '{{ var("pii_salt") }}'), 256)             as contact_hash,
    cast(null as date)                                               as date_of_birth,
    left(postal_code, 3)                                             as postal_code_prefix,
    cast(null as varchar)                                            as notes,
    segment,
    created_at
from {{ ref('customers') }}

The salt comes from a secret variable, never from the repository. Hashing with a shared salt keeps the column joinable across tables without exposing the value. Grants follow the policy: the default analyst role can select from gold.customers_masked, and only PRIVACY_OFFICE and FINANCE_OPS can read silver.customers.

The tests that guard certification

Certification is only worth something if it can be lost. The agent writes tests that fail when the table drifts from the policy:

• No unclassified columns. A test compares the columns in the warehouse with the columns tagged in YAML. A new column added upstream fails the build until someone classifies it.
• No PII in masked views. A test samples each masked view and fails if any value matches an email, phone or identifier pattern.
• Retention holds. A test fails if the oldest row in the table is older than retention_days plus one day of grace.
• Freshness and keys. The usual dbt unique, not_null and source freshness tests, as in the bronze, silver, gold example.
• Lineage is current. The lineage doc is regenerated from the dbt manifest on every build, and a check fails if the committed doc differs.

The retention job itself deletes rows, so it is treated with more care than a view. The agent writes the job and a dry-run report with row counts per table. A person approves the first run. After that it runs on its schedule, and every run posts its counts.

What the nightly check reports

The automation runs on the cron line 30 5 * * * after the nightly build. It is silent when nothing changed. When something did, the message is specific:

Nori  05:41
Governance check: 1 certified table no longer meets policy.

silver.customers: new column `marketing_consent_ip` appeared in last
night's load. It is unclassified, so `every_column_classified` fails.
Sampled values are IPv4 addresses. Proposed: pii_indirect, truncate to /24.

PR #517 adds the tag, updates customers_masked, and sets certified: false
until @maria.k approves. The column is not exposed in any masked view.

Every promotion, demotion and policy change is a merged pull request with a named approver. That history is your record of who certified what, under which version of the policy. For the wider schedule-and-webhook setup, see agentic data operations. To find data that never entered this pipeline at all, run the shadow data scan.

Frequently asked questions