Module trase.tools.aws.etag_cache

Caching of objects downloaded from S3.

Downloading an object writes its ETag - the hash that S3 keeps for every object - into a hidden "sidecar" file next to the downloaded file. Downloading

2026/downloaded/flows.csv

therefore also writes

2026/downloaded/.flows.etag.txt

On the next run we compare the sidecar with the ETag that S3 reports for the object: if they are the same then the local copy is up-to-date and the download can be skipped. A file with no sidecar is simply downloaded again.

Why record what S3 told us instead of hashing the local file? Because an ETag is only the MD5 of the contents for objects that were uploaded in a single request. Objects uploaded with the multi-part API - which is anything large, since both the AWS CLI and boto3 switch to multi-part uploads at 8MB by default - instead have an ETag of the form "-", which cannot be reproduced without knowing the part size that was used. (And an object encrypted with SSE-KMS has an ETag which is not a hash of its contents at all.) Recording the ETag at download time side-steps all of that: we never have to hash anything ourselves.

Functions

def delete_etag_sidecar(path) ‑> None

Forget the ETag recorded for a file, for example because it is being replaced.

def download_file_with_etag(client, bucket: str, key: str, path: str, version_id: str | None = None) ‑> None

Download an object from S3, recording its hash so that the download can be skipped next time (see :func:file_differs()).

The ETag is read before downloading: were the object to be replaced while we are downloading it, we would record the hash of the older object and simply download the file again on the next run. Recording the hash afterwards would risk the opposite - marking the contents we hold as up-to-date when they are not.

def etag_sidecar_path(path) ‑> str

The path of the hidden file which records the ETag of a downloaded object.

Example

>>> etag_sidecar_path("2026/downloaded/flows.csv")
'2026/downloaded/.flows.etag.txt'

def file_differs(client, bucket: str, key: str, path: str, version_id: str | None = None) ‑> bool

Determine whether the object at s3://BUCKET/KEY is different from a local file.

The comparison is made on content hashes: we compare the ETag that S3 reports for the object against the ETag recorded in the sidecar file which was written when the local file was downloaded (see :func:etag_sidecar_path()).

The byte size is compared first, which is both cheaper and catches a local file that was truncated or modified since it was downloaded.

If the local file does not exist, or has no sidecar recording where it came from, this is considered to be different from the remote and this function will return True.

def normalise_etag(etag: str | None) ‑> str | None

Strip the quotes that S3 wraps ETags in, e.g. '"d41d8cd9…"'.

def read_etag_sidecar(path) ‑> str | None

The ETag recorded for a downloaded file, or None if there isn't one.

def write_etag_sidecar(path, etag: str) ‑> None

Record the ETag of a file that was downloaded from S3.

A failure to write the sidecar is raised rather than swallowed: without it we have no record of which object the file is, and callers treat that as an error rather than silently work with data they cannot judge the freshness of.