Module `trase.tools.data_release_package.snapshot.cli`

CLI implementing the "snapshot" step for a data release package (see doc/Data-Release-Package.md).

Snapshotting ingests a package's data and metadata into the PostgreSQL database. Every package is snapshotted into a PostgreSQL schema of its own, named after the package folder, e.g. data_package_2026_indonesia_wood_pulp_v3_2_0. Within that schema the destination table for each file is fixed:

source file                         | destination table
------------------------------------|-------------------------------
data/spatial_metrics.data.txt       | spatial_metrics
data/supply_chains.data.txt         | supply_chains
spatial_layers/*.data.txt           | regions (unioned)
metadata/context.json               | supply_chains_context_metadata
metadata/spatial_metrics.json       | spatial_metrics_metadata
metadata/supply_chains_columns.json | supply_chains_columns_metadata

That mapping is spelled out, one call per file, in snapshot_package() below.

A file that is present is snapshotted; a file that is absent is simply skipped. Whether it should have been there is the job of the validation CLI (python -m trase.tools.data_release_package.validate), not of this one.

documentation/methodology.md is deliberately not snapshotted: Markdown documentation is for humans, and the database is no place for it.

Execution:

python -m trase.tools.data_release_package.snapshot <path>

A snapshot is a full replacement. An externally-published package should never be overwritten: publish a new version of it instead (see OVERWRITE_WARNING below).

The whole snapshot runs in a single transaction: the schema is created if needed, any tables already in it are dropped, and the package is written. Nothing is committed unless every table lands, so a failure leaves the previous snapshot's schema as it was.

Once it has committed, the package is declared as a source of the trase.earth dbt project, holding exactly the tables that were written. The declaration goes in models/trase_earth/_data_package_sources.yml, which holds the packages and nothing else and is written by this CLI alone (see dbt_sources). That declaration is how the pipeline knows the package is there and which parts of it there are to read, and it is a change to the Git working tree, to be committed by hand.

The script exits with a non-zero status code if the package is invalid or the snapshot fails.

Functions

def add_arguments(parser: argparse.ArgumentParser) ‑> None

def confirm_overwrite(package_dir: pathlib.Path, schema: str, tables: List[str], force: bool) ‑> bool

Asks before a snapshot already in schema is overwritten, saying what is at stake.

An empty (or absent) schema has nothing to overwrite, so nothing is asked. Otherwise the warning is printed either way, --force or not: someone overwriting a package should know what overwriting one means, whether or not they are being asked.

Args

package_dir : Path: The package about to be snapshotted, named in the warning.
schema : str: The destination schema.
tables : list: What is already in that schema, as snapshotted_tables() found it.
force : bool: True to overwrite without asking.

Returns

bool: True to go ahead with the snapshot, False if it should be abandoned.

def ingest_json_from_github(json_path: pathlib.Path, schema: str, table: str, connection)

Snapshots one metadata/*.json file, read straight from the Git checkout.

Args

json_path : Path: The metadata file to snapshot.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def ingest_parquet_from_s3(pointer_path: pathlib.Path, schema: str, table: str, connection)

Snapshots the Parquet file behind one *.data.txt pointer.

Args

pointer_path : Path: The pointer file naming the Parquet file in S3.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def ingest_spatial_layers(pointer_paths: List[pathlib.Path], schema: str, table: str, connection)

Snapshots every spatial_layers/*.data.txt pointer, unioned into the one table.

Args

pointer_paths : list: The pointer files, one per layer.
schema : str: The destination schema.
table : str: The destination table.
connection: The open connection to write on.

def main(args) ‑> bool

Snapshots the package at args.path into PostgreSQL.

Returns

bool: True if the package could not be snapshotted, otherwise False.

def report_snapshotted(schema: str, table: str) ‑> None

Reports a table that has been written but not yet committed.

def run() ‑> None

def snapshot_package(package_dir: pathlib.Path, schema: str, connection) ‑> List[str]

Snapshots every part of package_dir that is there.

This is the mapping from the specification to the database, written out one file at a time. Anything absent is skipped.

Args

package_dir : Path: The local package folder.
schema : str: The destination schema.
connection: The open connection to write on.

Returns

list: The tables that were written, which is what the package turned out to hold, and what its dbt source is then declared to have.

def snapshotted_tables(engine, schema: str) ‑> List[str]

The tables an earlier snapshot has left in schema, which a new one would drop.

Read on a connection of its own, before the snapshot's transaction is opened, so that we are not holding a transaction open while waiting for someone to answer the prompt below.

Args

engine: The engine to ask.
schema : str: The destination schema, which need not exist yet.

Returns

list: The tables in there, empty if the schema is empty or not there at all.

def update_dbt_sources(schema: str, tables: List[str], snapshotted_into_own_schema: bool) ‑> None

Declares the snapshotted package as a dbt source of the trase.earth pipeline.

The pipeline reads a package through the source of the same name, and builds its unions out of the tables that source declares, so this is what makes the snapshot visible to it (see dbt_sources).

A snapshot into a schema of someone's own choosing is left out of it: that is a scratch copy, and the sources file describes the packages the website is built from.