Module trase.tools.aws.upload

This module writes data files to disk and uploads them to AWS S3.

The functions here follow a deliberate "human in the loop" design: by default the file is written to disk and the user is told how to upload it themselves, rather than the upload happening automatically. This minimises the chance of accidentally overwriting data on S3, and gives the user a chance to inspect the data before it is uploaded.

Functions

def upload_file(client,
bucket: str,
key: str,
path: str,
printer=<bound method Logger.debug of <Logger trase.tools.aws.upload (WARNING)>>)

Uploads a file to S3.

Args

client

the boto3 S3 client to upload with

bucket

the s3 bucket that the file should be uploaded to

key

the s3 key that the file should be uploaded to

path

the location of the file on disk

def write_csv_for_upload(df: pd.DataFrame,
key: str,
path: str = None,
bucket: str = 'trase-storage',
do_upload: bool = None,
**pd_to_csv_kwargs)

Writes a CSV file and either indicates to the user how they should upload it to, S3 or actually performs the upload itself.

It is a very intentional design that, by default, the upload to S3 does not occur. We want the user to explicitly choose to do the upload to minimise the chance that they accidentally overwrite data on S3. By putting a "human in the loop" we also give the user a chance to inspect the data before upload.

Args

df

the Pandas dataframe that should be serialized

key

the s3 key that the dataframe should be uploaded to

path : optional

the (temporary) file location where the Pandas dataframe will be written to before uploading. The reason it is written to disk is to give the user the chance to inspect its contents before uploading. If not provided then a path will be generated in the operating system's temporary directory.

bucket : optional

the s3 bucket to upload the file to. Defaults to bucket defined in trase.config.settings.

do_upload : optional

whether to actually do the upload or just print to the user that they should do it themselves. If not provided then we will look for the presence of "–upload" in sys.argv to determine its value.

pd_to_csv_kwargs

keyword arguments that will be passed to pd.to_csv

def write_file_for_upload(key: str,
do_write,
path: str = None,
bucket: str = 'trase-storage',
do_upload: bool = None)

def write_geopandas_for_upload(gdf: gpd.GeoDataFrame,
key: str,
path: str = None,
bucket: str = 'trase-storage',
do_upload: bool = None,
driver: str = 'geojson',
**to_file_kwargs)

Writes a GeoDataFrame to disk and optionally uploads to S3.

Supported drivers: - "parquet" -> GeoParquet - "geojson" -> GeoJSON - "gpkg" -> GeoPackage

def write_json_for_upload(data,
key: str,
path: str = None,
bucket: str = 'trase-storage',
do_upload: bool = None,
**json_dump_kwargs)

def write_parquet_for_upload(df: pd.DataFrame,
key: str,
is_polars=False,
path: str = None,
bucket: str = 'trase-storage',
do_upload: bool = None,
**to_parquet_kwargs)