How to find shadow data: untracked tables, exports, and copies

Quick answer

Shadow data is any dataset your organization holds but does not track: a backup table from a migration, a CSV export in a bucket, a spreadsheet copy on a shared drive. To find it, a Nori agent lists everything in your warehouse, S3 buckets and shared drives, compares the list with your catalog, and reports what is missing. Each finding has a likely owner, PII flags, a last-access date and a keep, register or delete recommendation. Deletion always needs a person's approval.

Where shadow data comes from

Nobody creates shadow data on purpose. It is what is left after ordinary work:

• Backup and scratch tables. orders_backup_20240312, tmp_jl_test, customers_copy2. Made before a risky change, never dropped.
• Exports. A query result saved as CSV to a bucket for a vendor, a one-off analysis, or a support request.
• Copies outside the warehouse. The same export downloaded and uploaded to a shared drive as a spreadsheet.
• Orphaned pipelines. Tables still loaded nightly by a job whose owner left and whose consumers are gone.

These datasets cost storage, but the real problem is that your controls do not reach them. A masked view in the warehouse does nothing for an unmasked export in a bucket. Retention jobs skip tables they do not know about. You cannot answer a question about where customer data lives if a third of the copies are not in the catalog.

How to run the scan, step by step

1. Give the agent read-only access. A warehouse role that can read metadata and sample rows, an AWS role with s3:ListBucket and s3:GetObject on the buckets in scope, and read access to the shared drives. Add them to Nori as integrations scoped to the data agent. No delete permission is granted.
2. Point it at the catalog. The agent needs the list of tracked assets: your catalog's API or MCP server, a catalog export, or the dbt manifest.json if dbt is your catalog.
3. Ask in Slack. @nori scan the ANALYTICS and RAW databases, the s3://acme-data-* buckets and the Finance and Ops shared drives for datasets that are not in the catalog. Report to #data-governance.
4. It inventories each system. Tables and views from the warehouse metadata, objects from S3 listings or S3 Inventory reports, and CSV, Excel and Sheets files from the drive API.
5. It matches against the catalog. Anything with no catalog entry is a candidate. For each one it looks for a tracked dataset with the same columns, to tell copies from originals.
6. It profiles the candidates. Size, row count, creation date, last access, a sample of values checked for PII patterns, and owner signals.
7. It writes the report with a recommendation per dataset and posts the summary. Nothing is changed.
8. People decide. Owners reply per row. Register and keep decisions become pull requests. Delete decisions follow the approval path below.

Worked example: the inventory query

For Snowflake, the agent builds the candidate list from account usage views and an export of the catalog loaded as governance.catalog_assets. Last access comes from access_history, which records the tables each query read. Other warehouses have equivalent metadata, and the agent adapts the query.

with last_read as (
    select
        obj.value:objectName::string   as fq_name,
        max(ah.query_start_time)       as last_read_at,
        count(distinct ah.user_name)   as reader_count
    from snowflake.account_usage.access_history ah,
         lateral flatten(input => ah.base_objects_accessed) obj
    where ah.query_start_time >= dateadd('day', -365, current_timestamp())
    group by 1
)

select
    t.table_catalog || '.' || t.table_schema || '.' || t.table_name  as fq_name,
    t.table_owner,
    t.row_count,
    round(t.bytes / power(1024, 3), 2)   as size_gb,
    t.created,
    t.last_altered,
    r.last_read_at,
    coalesce(r.reader_count, 0)          as readers_365d
from snowflake.account_usage.tables t
left join last_read r
    on r.fq_name = t.table_catalog || '.' || t.table_schema || '.' || t.table_name
left join governance.catalog_assets c
    on upper(c.fq_name) = t.table_catalog || '.' || t.table_schema || '.' || t.table_name
where t.deleted is null
  and t.table_schema <> 'INFORMATION_SCHEMA'
  and c.fq_name is null                  -- not in the catalog
order by t.bytes desc nulls last

Account usage views lag real time by up to a few hours and access_history keeps one year, so a null last_read_at means no read in the past 365 days, not never. The agent states that limit in the report instead of writing "never accessed".

For S3 it lists objects with aws s3api list-objects-v2, or reads S3 Inventory reports for large buckets, and keeps data file types: CSV, Parquet, JSON, Excel and database dumps. For shared drives it lists spreadsheet and CSV files with their owner, last modifier and sharing settings.

How the agent guesses an owner and flags PII

An untracked dataset has no owner field by definition, so the agent collects evidence and says how strong it is:

• Warehouse: the role that created the table, the users who queried it most in the past year, and initials or team names in the table name.
• S3: bucket and object tags, the prefix path, and the principal in CloudTrail PutObject events where data events are logged.
• Shared drives: the file owner and last modifier.
• Copies: if the columns and row count match a tracked table, the tracked table's owner is the likely owner of the copy.

PII flags come from column names and a sample of values checked against patterns for emails, phone numbers, card numbers and national identifiers. If you already keep a policy file as in the data governance pipeline example, the agent uses its classes. A flag is a reason to look, not a legal classification.

Example report and what happens to each row

Each recommendation leads to a different, reviewable action:

• Register. The agent opens a pull request that adds the dataset to the catalog source with owner, description and classification. The owner reviews it.
• Keep. The decision and its reason are recorded so the dataset is not reported again next month.
• Delete. The agent never deletes during a scan. After the owner replies approve delete in the thread, it prepares the change: for a table, a pull request with the drop table statement and the recovery window; for a file, the exact object path. A person with delete rights runs or merges it. Where you allow it, the first step is a rename or a move to a quarantine prefix for 30 days.

Owner guesses can be wrong. The report is addressed to the likely owner as a question, and a reply of "not mine" sends the row back with that person excluded.

Run it every month

Shadow data comes back, so a single scan is only a snapshot. A Nori automation on 0 6 1 * * reruns the scan on the first of each month and reports only what is new, what is still undecided, and how much was registered or removed since last time. Past decisions are stored in shared memory, so a dataset marked keep stays quiet.

The monthly scan is one part of a wider routine. Agentic data operations covers the health checks and cleanup that run next to it, and the bronze, silver, gold example shows how to give new data a tracked home from the first load.

Frequently asked questions