Module trase.tools.data_release_package.validate.validators.data

Validates the data behind the *.data.txt pointers.

Everything else in validate reads only the local package folder. These checks follow a pointer to the Parquet file it names in S3, so they are marked @accesses_s3 and are left out by --skip-s3.

Functions

def row_count(s3_url: str) ‑> int

The number of rows in a Parquet file in S3.

Only the file's footer is read - the row count is in its metadata - so this costs the same whether the file is a megabyte or a hundred gigabytes.

def validate_data_files_are_not_empty(package_dir: pathlib.Path) ‑> Iterator[str]

Every file a package points at must have rows in it.

A pointer to an empty file is the failure that is hardest to see from the outside: the package validates, snapshots and builds, and the context simply disappears from the website. It is usually a data file that was written from a query that returned nothing, or a pointer left behind pointing at a path that has since been rewritten.