Data Validation Tool

How we prove that a migrated dataset matches its source. This section collects what Aliz has learned about migration validation tooling: the market landscape, and a hands-on runbook for Google's Data Validation Tool (DVT) validating Oracle and PostgreSQL sources against BigQuery.

When you need this

Every data migration ends with the same question from the client: is the data in the new system the same as in the old one? Row counts alone do not answer it. A defensible answer covers four layers:

Layer	Question it answers	Typical check
Schema	Do the same columns exist with compatible types?	Column existence and type mapping
Column	Do the tables agree in bulk?	Row counts, SUM / MIN / MAX / AVG per column
Row	Does every row match?	Per-row hash compared on the primary key
Business rule	Do the numbers the client cares about match?	Custom SQL on both sides, compared on the output

Use this section when you are planning or executing a migration into BigQuery (or any other GCP target) and need to pick a tool, size the effort, and run the validations.

How to use this section

"Which tool should we use?" — Read Market options. It compares seven candidates on supported sources, hosting, licence, and functionality.

"We chose DVT, how do I run it?" — Follow the DVT runbook. It is a validated end-to-end walkthrough from install to persisted results, including the mistakes we made so you do not repeat them.

"How big a machine does DVT need?" — See Memory, sizing, and large tables in the runbook for memory per row, partitioning, and throughput figures.

Our recommendation

For migrations into BigQuery, start with Google's Data Validation Tool (DVT). It is open source, actively maintained, self-hosted, and supports the widest set of sources we meet on migration projects (Oracle, Teradata, PostgreSQL, SQL Server, Hive, Spanner, Snowflake, and more). Aggregates and per-row hashes are computed inside each engine and compared client-side: column validation moves only a handful of scalars at any scale, while row validation holds one hash per row in memory and scales out by partitioning the table across workers. Results land in a BigQuery table you can report on. The runbook shows it working end to end.

Reach for something else when the fit is obvious: dbt-audit-helper if the whole pipeline is already dbt, datacompy if both sides are already DataFrames in one process, Datafold Cloud if the client wants a SaaS UI with lineage. See How to choose for details.

What makes validation hard

These are the properties a validation tool must handle. Use them as evaluation criteria when a client proposes a tool of their own.

Challenge	Why it matters
Scale	Comparing 100 rows is trivial. Comparing 100 million rows requires sophisticated algorithms and significant compute.
Schema evolution	Production data changes schemas. Good diff tools handle column additions, renames, and type changes gracefully.
Floating point precision	0.1 + 0.2 does not exactly equal 0.3. Diff tools need tolerance thresholds for numeric comparisons.
Sort order	Two datasets might be identical but in different orders. Tools need primary keys to match rows correctly.
NULL handling	Is NULL equal to NULL? Different systems have different semantics.
Performance	Naive comparison is O(n²). Large datasets need hash-based or merge-join approaches.
Output readability	"1 million rows differ" is not actionable. Good diffs show sample differences and summary statistics.

Source material

The pages in this section were ingested from two working documents in September 2026:

• Migration validation tooling (Google Sheet) — the tool comparison matrix
• DVT Data Validation Runbook — Oracle / PostgreSQL to BigQuery (Google Doc) — the end-to-end test log