Module trase.tools.etl.context

Classes

class Context (year: int,
data_directory: str,
extra: dict | None = <factory>,
pre_extracted_data: dict | None = None,
skip_cache: bool | None = None)

Simple container for useful information about a model.

This object can be passed around the preparation script, run script, or elsewhere, to provide global variables for a model.

If the variable is really a hard-coded constant, consider putting it in a file like constants.py instead.

Args

year

the year for which the model is related to. This will be used, for example, in conjunction with the data_directory argument to determine where to store input and output data for the model.

data_directory

path to a directory on the local file system which should be used to store input and output data for the model.

extra : optional

a dictionary of extra metadata for the model, such as a model parameter.

pre_extracted_data : optional

dataframes to pass to the preparation files.

skip_cache : optional

if True, do not use the cache of files previously downloaded from S3: download them again instead. If False, always use the cache. If None (the default) then the configuration setting TRASE_ETL__SKIP_CACHE decides, which itself defaults to False.

Instance variables

var data_directory : str

var extra : dict | None

var pre_extracted_data : dict | None

prop should_skip_cache : bool

Whether files should be downloaded from S3 without consulting the cache.

This resolves the skip_cache argument against the Trase configuration: see the documentation of that argument.

var skip_cache : bool | None

var year : int

Methods

def replace(self, **changes)

Return a copy of the instance except with some fields overwritten.

This method is identical to dataclasses.replace except that it will handle overwriting only a subset of keys of extra:

# we have two pieces of extra information: "complex" and "percent"
old = Context(2013, "/tmp/", extra=dict(complex=True, percent=10))

# we alter percent only
new = old.replace(extra=dict(percent=50))

# percent has updated, but complex is unchanged
new.extra["complex"]  # True
new.extra["percent"]  # 50

If you do not wish to have this behaviour then simply use dataclasses.replace