Module trase.tools.etl

Extract-transform-load from a variety of sources into Pandas dataframes.

The ETL tool handles:

  • Loading data into Pandas from a variety of sources (AWS S3, PostgreSQL database, etc.) and in a variety of formats (XLS, CSV)
  • Only downloading the data once, yet still re-downloading it when the source data has changed
  • Allows you to do some simple pre-processing, such as filtering out rows, or altering some data
  • Storing this data locally in a standard directory layout ("original", "processed", etc.)

The ETL tool is primilary targeted for use with SEI-PCS models; however, it is a standalone package.

Caching of files downloaded from S3

A file that is downloaded from S3 is written to the downloaded directory of the model, for example 2026/downloaded/flows.csv. Alongside it we write a hidden sidecar file which records the hash (the S3 ETag) of the object that was downloaded:

2026/downloaded/.flows.etag.txt

On the next run, the hash that S3 reports for the object is compared against the hash in the sidecar file. If they are the same then the object has not changed and the download is skipped; if they differ then the object is downloaded and processed again.

A file which has no sidecar - because it was downloaded before sidecars existed, for example - is simply downloaded again.

Note that the hash is recorded rather than computed from the local file. This is because an ETag is only the MD5 of the contents for objects which were uploaded to S3 in a single request: large objects are uploaded in multiple parts and get an ETag of the form "<md5 of the concatenated part md5s>-<number of parts>" instead, which cannot be reproduced without knowing the part size that was used. Recording what S3 reports means we never have to hash an object ourselves.

Skipping the cache

To ignore the cache and always download objects again, either pass skip_cache in Python:

supplychain.preparation(skip_cache=True)

or set it in the configuration (see Configuration in Trase):

TRASE_ETL__SKIP_CACHE=true

The Python argument takes precedence over the configuration setting. skip_cache can also be passed to the SupplyChain constructor, or set directly on a Context when preparation classes are used without a SupplyChain.

Sub-modules

trase.tools.etl.context
trase.tools.etl.exceptions
trase.tools.etl.pandas_wrapper
trase.tools.etl.processors
trase.tools.etl.source
trase.tools.etl.utilities