Module trase.tools.data_release_package.export

Build the files of a data release package.

Everything needed to produce a data release package lives here; cli and validators alongside it check a package once it has been produced. See doc/Data-Release-Package.md for the specification itself.

The code here is still specific to the Côte d'Ivoire cocoa v1.2.0 package: the constants below name that context, that model and those metrics. It is gathered in one module rather than scattered through the package's script folders so that it can be generalised in one place when the other supply chain contexts are exported the same way. What is context-specific is the block of constants at the top; the functions under it are already close to generic.

The thin scripts that call in here are the entry points, one per published file:

trase/data/cote_divoire/cocoa/data_packages/cote_divoire_cocoa_v1_2_0/
    supply_chains.py   -> supply_chains.parquet
    spatial_metrics.py -> spatial_metrics.parquet

trase/data/cote_divoire/spatial/boundaries/trase_earth/
    country.py, region.py, district.py, department.py -> <layer>.parquet

The package's metadata/*.json are written into Git rather than to S3, by running this module directly:

python -m trase.tools.data_release_package.export

Functions

def build_and_publish(query, s3_key, script_path, validate=None)

Run one query, write the result to Parquet, and publish it to S3.

Without --upload on the command line the file is written to a local temp directory and the upload command printed instead, which is a safe dry run; that is handled by write_file_for_upload. validate, if given, is called with the connection and the written file before it can be uploaded.

def build_and_publish_data_file(query, filename, script_path)

Publish one of the package's data/ files.

def build_and_publish_layer(node_type_slug, filename, script_path)

Publish one of the boundary layers the package's spatial_layers/ point at.

def build_context()

def build_spatial_metrics()

The metric metadata, keyed by the metric's released column name.

The seed's backend_name is the database metric name (COCOA_AREA); the specification wants backend_name to name a column of the released Parquet file (cocoa_area_ha), so it is translated on the way out.

def build_supply_chains_columns()

def convert(field, value)

Turn one seed cell into its JSON value.

The seeds are CSV, so everything arrives as text: an empty cell means "not set" rather than an empty string, and the numeric and boolean fields have to be given their real types.

def read_seed(path)

def select(row, fields)

def spatial_layer_query(node_type_slug)

The query for one spatial layer, in specification column order.

views.regions stores geometry as GeoJSON text (_geometry) and both year bounds as double precision, so they are converted here to the geometry and BIGINT types the specification asks for. Rows whose geometry is null are kept: the specification does not require a layer row to have geometry, and an entity is worth publishing whether or not Trase holds a boundary for it.

def spatial_metrics_query()

Pivot the long spatial_metrics table into one column per metric.

The source is long: one row per metric per region per year, for every level the country publishes. (region_trase_id + metric_name + year + country_slug + commodity_slug + node_type_slug) is unique there, so at most one row feeds each metric of each output row and max(…) FILTER (…) is a pivot, not an aggregation. node_type_slug is taken with any_value rather than grouped on, so that a trase_id can never produce two rows for one year and break the (trase_id + year) primary key.

def supply_chains_query()

Trade flows for the Sankey / exposures, one row per flow.

def validate_written_layer(connection, path)

Check a written spatial layer against the specification.

Raises rather than letting a non-conforming file reach S3: the columns, their order and their types must match the specification exactly, and the year ranges of any one trase_id may not overlap. views.regions holds one row per (trase_id, year_start, year_end), so a single entity can legitimately appear more than once – with adjacent, non-overlapping ranges – when its name, parent or geometry changed part-way through.

def write_json(content, path)

def write_metadata_files()

Write the package's three metadata/*.json files into the package folder in Git.

Re-run this and commit the result whenever the seeds change. It fails rather than writing a package whose metadata and data disagree.