Module trase.tools.etl_internal.test.test_processors
Functions
def context(tmp_path)def pretend_the_data_was_processed(processor)-
Leave behind what a completed run of the processor would leave behind.
The processed file itself is faked, since these tests are about the caching and not about the reading and writing of dataframes.
def s3()def test_a_changed_object_is_downloaded_again(context, s3)def test_a_changed_object_is_processed_again(context, s3)-
A changed object must not merely be downloaded again: the data has to be processed again too, otherwise the prepared file is left describing the old object.
def test_a_downloaded_object_is_not_downloaded_again(context, s3)def test_a_missing_hash_is_fatal(context, s3)-
We would be unable to judge whether the prepared file is up-to-date, so we stop rather than carry on and risk using stale data
def test_an_unchanged_object_is_not_processed_again(context, s3)def test_extract_records_the_hash_of_the_object(context, tmp_path, s3)def test_pre_extracted_data_has_no_hash(context, s3)-
Data handed to us by DBT was never downloaded from S3, so there is no hash
def test_skip_cache_can_be_configured(context, s3, monkeypatch)def test_skip_cache_processes_the_data_again(context, s3)def test_skip_cache_skips_the_cache(context, s3)def test_the_hash_of_the_source_is_part_of_the_metadata(context, s3)def without_configured_skip_cache(monkeypatch)
Classes
class Assets (*args, client=None, fallback_bucket='trase-storage', **kwargs)-
Ancestors
Class variables
var columnsvar keyvar outname
Inherited members
class FakeS3-
A mock S3 client serving a single object, which can be replaced.
The ETags here are deliberately not hashes of the contents: S3 gives a large object an ETag which cannot be reproduced from its contents, and we only ever compare the ETag we recorded with the one S3 reports.
Methods
def put(self, contents: bytes, etag: str)