Maintained data-diff tool for large CSV and Parquet files with data-version-control integration

dev tool real project •• multiple requests

Data engineers want to diff big tabular files (CSV, Parquet) the way git diffs code, seeing exactly which rows, columns, and values changed between two versions, with hooks into DVC. The leading open-source option, Datafold's data-diff, was archived in 2024, and the survivors either choke on Parquet or load everything into pandas. The core value is a fast, file-first, schema-drift-tolerant diff that slots into a version-controlled data workflow.

builder note

DuckDB already does the heavy lifting of reading the files, so the unsolved part is the presentation and the workflow fit (a column-aware, schema-drift-tolerant diff that drops into a DVC or git pre-commit hook), not the comparison math itself.

landscape (3 existing solutions)

Tabular diffing splintered into abandoned (data-diff), small-file CLI tools (daff), and hand-rolled SQL or pandas. Nothing today combines large-Parquet scale, a readable row, column, and value diff, and awareness of a data-version-control flow.

Datafold data-diff Archived and read-only as of 2024-05-17 with a last release in February 2024. It was also database-to-database oriented rather than file-first, so it never cleanly handled a loose pile of Parquet on disk.

daff Supports only CSV, TSV, SQLite, and ndjson (no Parquet), is tuned for small-to-medium tables that share a common origin, and integrates with git as a diff handler but not with DVC.

DuckDB (as a DIY workaround) Can read Parquet at scale and express a comparison in SQL, but it is a general engine with no purpose-built diff output, no schema-drift handling, and no version-control awareness, so every team rebuilds the same ad hoc query.

sources (1)

hn https://news.ycombinator.com/item?id=48431700 "still no good 'data diff' tool for a big pile of CSV or Parquet" 2026-06-07

data-engineeringparquetdvcdiffingcli