Module trase.tools.etl_internal.processors

Classes

class MissingSourceHash (*args, **kwargs)

We have no record of which object in S3 a downloaded file came from

Ancestors

  • builtins.Exception
  • builtins.BaseException
class Preprocessor (*args, client=None, fallback_bucket='trase-storage', **kwargs)

Ancestors

Subclasses

Methods

def should_rerun(self, args)

Inherited members

class S3Mixin (*args, client=None, fallback_bucket='trase-storage', **kwargs)

Ancestors

Subclasses

Class variables

var bucket
var version_id

Instance variables

prop full_s3_path
prop original_extension

Methods

def construct_metadata(self, args)

Include the hash of the source object in the metadata of the prepared file.

Without this, replacing an object in S3 would cause it to be downloaded again but not processed again: by the time should_rerun is consulted the new object has already been downloaded, so nothing it looks at has changed and the prepared file is left describing the old object.

The hash is read from the sidecar file rather than from S3, since the object has already been downloaded (and the sidecar written) by the time this is called.

Raises

MissingSourceHash
if there is no sidecar file. This should be impossible - a file with no sidecar is downloaded again, which writes one - so it means the sidecar could not be written. Rather than carry on and produce a prepared file whose freshness nothing can judge, we stop.
def extract(self, path)
def s3_key(self)
def should_reextract(self, path) ‑> bool

This function checks whether we are able to skip re-downloading the object from S3, by comparing the hash (ETag) that S3 reports for the object with the hash we recorded when we last downloaded it.

This cache can be disabled - so that the object is always downloaded again - either by setting skip_cache on the context (see Context) or by setting TRASE_ETL__SKIP_CACHE in the Trase configuration.

class TextPreprocessor (*args, client=None, fallback_bucket='trase-storage', **kwargs)

Ancestors

Methods

def should_rerun(self, args)

Inherited members