Module trase.tools.data_release_package.validate.validators.data

Validates the data behind the *.data.txt pointers.

Everything else in validate reads only the local package folder. These checks follow a pointer to the Parquet file it names in S3, so they are marked @accesses_s3 and are left out by --skip-s3.

Functions

def context_country(package_dir: pathlib.Path)

The country named by metadata/context.json, if it names one.

The context calls it country_of_production; the layers publish the same upper-case name in their country column.

def distinct_combinations(s3_url: str, columns: Sequence[str]) ‑> List[tuple]

Return the distinct combinations of values across the given columns.

For example, given:

+----------+-------+
| type     | level |
+----------+-------+
| building |     1 |
| bridge   |     2 |
| building |     2 |
| building |     1 |
+----------+-------+

This returns:

[
    ("building", 1),
    ("bridge",   2),
    ("building", 2),
]

def distinct_values(s3_url: str, columns: Sequence[str]) ‑> Dict[str, List]

Return the distinct values of each column independently.

For example, given:

+----------+-------+
| type     | level |
+----------+-------+
| building |     1 |
| bridge   |     2 |
| building |     2 |
+----------+-------+

This returns:

{
    "type": ["building", "bridge"],
    "level": [1, 2],
}

def row_count(s3_url: str) ‑> int

The number of rows in a Parquet file in S3.

Only the file's footer is read - the row count is in its metadata - so this costs the same whether the file is a megabyte or a hundred gigabytes.

def validate_data_files_are_not_empty(package_dir: pathlib.Path) ‑> Iterator[str]

Every file a package points at must have rows in it.

A pointer to an empty file is the failure that is hardest to see from the outside: the package validates, snapshots and builds, and the context simply disappears from the website. It is usually a data file that was written from a query that returned nothing, or a pointer left behind pointing at a path that has since been rewritten.

def validate_spatial_layers_carry_the_country(package_dir: pathlib.Path) ‑> Iterator[str]

Every row of every spatial layer names the package's country of production.

A package covers one supply chain context, so country_trase_id and country hold one value for the whole package: the same in every row, in every layer, and never null. Consumers union the layers of many packages into a single entity table, where those columns are all that says which country an entity belongs to, so a layer that leaves them empty - or one layer of six built before the columns existed - is a package that looks fine until its entities lose their country.

def validate_spatial_level_consistency(package_dir: pathlib.Path) ‑> Iterator[str]

Validates that spatial layer levels are populated and consistent:

1. Completeness: No row has a null level.

2. Consistency: A specific node type within a country (e.g., 'kabupaten' in Indonesia) has the same level across all files and years.