Module `trase.tools.data_release_package.snapshot.cli`

CLI implementing the "snapshot" step for a data release package (see doc/Data-Release-Package.md).

Snapshotting ingests a package's data and metadata into the PostgreSQL database. Every package is snapshotted into a PostgreSQL schema of its own, named after the package folder, e.g. data_package_2026_indonesia_wood_pulp_v3_2_0. Within that schema the destination table for each file is fixed:

source file                         | destination table
------------------------------------|-------------------------------
data/spatial_metrics.data.txt       | spatial_metrics
data/supply_chains.data.txt         | supply_chains
spatial_layers/*.data.txt           | regions (unioned)
metadata/context.json               | supply_chains_context_metadata
metadata/spatial_metrics.json       | spatial_metrics_metadata
metadata/supply_chains_columns.json | supply_chains_columns_metadata

That mapping is spelled out, one call per file, in snapshot_package() below.

A file that is present is snapshotted; a file that is absent is simply skipped. Whether it should have been there is the job of the validation CLI (python -m trase.tools.data_release_package.validate), not of this one.

documentation/methodology.md is deliberately not snapshotted: Markdown documentation is for humans, and the database is no place for it.

Execution:

python -m trase.tools.data_release_package.snapshot <path>

The package is validated first, then the whole snapshot runs in a single transaction: the schema is created if needed, any tables already in it are dropped, and the package is written. Nothing is committed unless every table lands, so a failure leaves the previous snapshot's schema as it was.

Once it has committed, the package is declared as a source of the trase.earth dbt project, holding exactly the tables that were written. The declaration goes in models/trase_earth/_data_package_sources.yml, which holds the packages and nothing else and is written by this CLI alone (see dbt_sources). That declaration is how the pipeline knows the package is there and which parts of it there are to read, and it is a change to the Git working tree, to be committed by hand.

The script exits with a non-zero status code if the package is invalid or the snapshot fails.

Functions

def add_arguments(parser: argparse.ArgumentParser) ‑> None

def ingest_json_from_github(json_path: pathlib.Path, schema: str, table: str, connection)

Snapshots one metadata/*.json file, read straight from the Git checkout.

Args

json_path : Path: The metadata file to snapshot.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def ingest_parquet_from_s3(pointer_path: pathlib.Path, schema: str, table: str, connection)

Snapshots the Parquet file behind one *.data.txt pointer.

Args

pointer_path : Path: The pointer file naming the Parquet file in S3.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def ingest_spatial_layers(pointer_paths: List[pathlib.Path], schema: str, table: str, connection)

Snapshots every spatial_layers/*.data.txt pointer, unioned into the one table.

Args

pointer_paths : list: The pointer files, one per layer.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def main(args) ‑> bool

Snapshots the package at args.path into PostgreSQL.

Returns

bool: True if the package could not be snapshotted, otherwise False.

def report_snapshotted(schema: str, table: str) ‑> None

Reports a table that has been written but not yet committed.

def run() ‑> None

def snapshot_package(package_dir: pathlib.Path, schema: str, connection) ‑> List[str]

Snapshots every part of package_dir that is there.

This is the mapping from the specification to the database, written out one file at a time. Anything absent is skipped.

Args

package_dir : Path: The local package folder.
schema : str: The destination schema.
connection: The open connection to write on.

Returns

list: The tables that were written, which is what the package turned out to hold, and what its dbt source is then declared to have.

def spatial_layer_pointers(package_dir: pathlib.Path) ‑> List[pathlib.Path]

Every spatial_layers/*.data.txt of the package, in a stable order.

def update_dbt_sources(schema: str, tables: List[str], snapshotted_into_own_schema: bool) ‑> None

Declares the snapshotted package as a dbt source of the trase.earth pipeline.

The pipeline reads a package through the source of the same name, and builds its unions out of the tables that source declares, so this is what makes the snapshot visible to it (see dbt_sources).

A snapshot into a schema of someone's own choosing is left out of it: that is a scratch copy, and the sources file describes the packages the website is built from.