Data Quality Scans

Data Quality scans help to check data for certain metrics, like completeness, uniqueness, validity, NULLs, consistency, or any custom rule you might need, like the length of the string, etc.

Masthead can integrate your own data quality scans or use Dataplex, a native Google Cloud solution, to power and execute the data quality rules, which are, in essence, SQL queries. The following sections detail both options.

Data scans with Dataplex

What data does Masthead query with Dataplex?

None. Masthead doesn’t request permissions, nor does it read or edit client data. You should grant read-only permissions to Dataplex, a Google Cloud native service. Masthead only helps you create and manage rules in Dataplex.

Is Dataplex available automatically?

You can enable this feature with a single click. First, activate the Google Cloud Dataplex API in the connected Google Cloud project. The Masthead UI provides a link to the API under the Data Quality tab. Additionally, you must manually create rules for each table you want to monitor. Once you create and schedule the rules, they execute automatically.

Which metrics can you monitor using Dataplex?

This depends on each business and the metrics that are crucial for it. Data Quality rules schedule SQL queries to run. There are pre-set rules for aspects like completeness, accuracy, consistency, validity, uniqueness, and null checks.

However, you can monitor any metric within any dimension using a custom SQL rule.

Why integrate Masthead with Dataplex?

Dataplex combined with Masthead Data Quality is a SQL-based solution, in contrast to Masthead Monitors, which rely on logs and metadata. Unlike Dataplex, Masthead Monitors provide automated anomaly detection across the entire data platform by identifying anomalies in time-series tables and errors in pipelines in real-time. Additionally, it doesn’t increase cloud costs because it doesn’t run SQL queries.

Dataplex is the optimal solution for Google Cloud users to implement data quality checks. The benefits include:

• It’s 100x cheaper. Pay only for SQL execution, which helps avoid the 100X additional costs associated with purchasing another SQL-first solution plus to compute costs for executed queries.
• The safest approach. Doesn’t expose data to a third-party vendor, which also eliminates the need to spend time with security and legal teams for onboarding a third-party tool.
• No additional vendor lock-in.

Custom data scans

Custom data scans bring your own data quality rule checks into Masthead. You publish data quality check results as a BigQuery table that holds one numeric metric per table and period. Masthead reads the table on a schedule, learns the patterns behind every metric, and raises a data quality incident when a value falls outside the forecasted range.

Caution

Custom data scans are available only on app.mastheadata.com. The UK region doesn’t support them.

Set up a data scan

• With an AI agent: The data scans skill walks your agent through the setup. It helps you write the metric query and checks it with your credentials. After you confirm each step, it creates the scan table and the scheduled query that keeps it current, grants Masthead access, and creates the scan through the Masthead MCP server.
• By hand: Create the scan table and its refresh, grant access as the following sections describe, and create the scan with the create_data_scan MCP tool.

You can’t create scans in the Masthead UI. Review them on the Data Quality page.

Table columns

Column	Type	Description
table_reference	STRING	The monitored table the metric describes, as project.dataset.table. Masthead attaches incidents to this table.
rule_name	STRING	The metric name, for example null_email_percent. It becomes the incident’s metric name, so keep it stable.
timestamp	TIMESTAMP	The start of the period the value describes, for example TIMESTAMP_TRUNC(updated_at, DAY).
value	INT64, FLOAT64, NUMERIC, or BIGNUMERIC	The metric value.

• Write one row per table_reference, rule_name, and period.
• Write a row for every metric in every period, including zero values. Masthead flags a period without a row as missing data.
• Don’t write NULL in any column. Masthead rejects a table whose rows contain NULLs.
• Keep at least 14 days of history. The first run analyzes the last 14 days.
• Express ratios as percentages (0–100) rather than fractions (0–1). Metrics with small values get wide expected ranges.
• Store aggregated metrics only, never raw rows.

Keep the table current

Add each closed period to the table before Masthead checks it, for example with a BigQuery scheduled query, dbt, or Dataform. Masthead checks a period after its processing delay ends, so schedule the refresh to finish before then.

Example table that tracks the daily share of customers without an email address:

1. Create the table with the last 30 days of history:

   CREATE TABLE `my-project.masthead_dq.customers_null_email` AS
   SELECT
     'my-project.crm.customers' AS table_reference,
     'null_email_percent' AS rule_name,
     TIMESTAMP_TRUNC(updated_at, DAY) AS `timestamp`,
     100 * COUNTIF(email IS NULL) / COUNT(*) AS value
   FROM `my-project.crm.customers`
   WHERE updated_at >= TIMESTAMP(DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
     AND updated_at < TIMESTAMP(CURRENT_DATE())
   GROUP BY table_reference, rule_name, `timestamp`

2. Schedule a query that adds the previous day, once a day after midnight UTC:

   MERGE `my-project.masthead_dq.customers_null_email` AS t
   USING (
     SELECT
       'my-project.crm.customers' AS table_reference,
       'null_email_percent' AS rule_name,
       TIMESTAMP_TRUNC(updated_at, DAY) AS `timestamp`,
       100 * COUNTIF(email IS NULL) / COUNT(*) AS value
     FROM `my-project.crm.customers`
     WHERE updated_at >= TIMESTAMP(DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
       AND updated_at < TIMESTAMP(CURRENT_DATE())
     GROUP BY table_reference, rule_name, `timestamp`
   ) AS s
   ON t.table_reference = s.table_reference AND t.rule_name = s.rule_name AND t.`timestamp` = s.`timestamp`
   WHEN MATCHED THEN UPDATE SET value = s.value
   WHEN NOT MATCHED THEN INSERT ROW

Grant access

Keep scan tables in a dedicated dataset, for example masthead_dq, in the same location as the source data. Masthead reads that dataset only.

Grant the Masthead service account read access to the dataset:

GRANT `roles/bigquery.dataViewer` ON SCHEMA `my-project.masthead_dq`
TO "serviceAccount:masthead-quality-checks@masthead-prod.iam.gserviceaccount.com";

How scans run

• Frequency: hourly, every 6 hours, every 12 hours, or daily.
• Processing delay: after a period ends, Masthead waits before checking it, so late data can arrive. The default is half the period, up to 12 hours.
• Trend and seasonality: each metric gets its own model. Masthead detects the seasonal cycle, such as a weekly pattern in daily data, once the metric has two full cycles of history.
• Late and corrected data: every run rechecks the last 3 periods, so rows that arrive late or get corrected update the result.
• Adapts over time: every run retrains the model on recent history (60 days for daily scans), so the expected range follows a metric as it grows or shifts.
• First run: starts as soon as you create the scan, analyzes the last 14 days, and doesn’t send notifications. Later anomalies raise data quality incidents that follow your alert settings.
• Cost: Masthead pays for the queries that read the scan table. The queries that refresh it run in your project.

Review results

The Data scans tab of the Data Quality page lists every data scan with its latest result. Each scan’s page shows the results per period, metric values against their expected ranges, the scan configuration, and the monitored tables.

See results in the context of your data assets.

FAQ

• Is data sampling supported?

  Yes, to make data quality checks more cost-efficient. You can filter it like any SQL query, based on data dimensions.

  Data Quality with Dataplex has an easy-to-use UI-based setting that allows you to limit the scope and sample the data queried.

• How much does it cost to run data quality checks?

  Masthead itself doesn’t charge for running data quality checks. Put simply, customers pay only for the execution of rules.

  The cost of each data quality check is equivalent to the data processed in BigQuery at the chosen data location and the selected pricing plan for the particular Google Cloud project.

  Dataplex charges based on pay-as-you-go usage. The cost to run data quality checks aligns with the Dataplex premium processing pricing.

• After creating a rule in Masthead Data Quality with Dataplex, does it appear in Dataplex in Google Cloud?

  Yes, once you create a rule in Masthead, it appears in the Dataplex UI, and vice versa. You can modify or delete rules created in Masthead within Dataplex at any time.