S3ObjectStore#

class composer.utils.S3ObjectStore(bucket, prefix='', region_name=None, endpoint_url=None, aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None, client_config=None, transfer_config=None)[source]#

Utility for uploading to and downloading from an S3-compatible bucket using boto3.

Note

Since boto3 has automatic retry handling, transient errors are not re-raised with ObjectStoreTransientError. By default, the โ€œLegacy retry modeโ€ makes at most five attempts for operations that failed due to transient errors. This setting can be customized via the retries key in client_config.

See the guide to retries for additional information.

Warning

For best security practices, it is recommended to set credentials via environment variables or config files. Never hard-code credentials via the aws_access_key_id, aws_secret_access_key, or aws_session_token arguments.

See guide to credentials for more information.

Parameters

  • bucket (str) โ€“ The bucket name.

  • prefix (str) โ€“ A path prefix such as folder/subfolder/ to prepend to object names. Defaults to โ€˜โ€™.

  • region_name (str, optional) โ€“ The region name. Must be specified if not available in a config file or environment variables. Defaults to None.

  • endpoint_url (str, optional) โ€“ The URL to an S3-Compatible object store. Must be specified if using something other than Amazon S3, like Google Cloud Storage. Defaults to None.

  • aws_access_key_id (str, optional) โ€“ The access key id. Must be specified if not available in a config file or environment variables. Defaults to None.

  • aws_secret_access_key (str, optional) โ€“ The secret access key. Must be specified if not available in a config file or environment variables. Defaults to None.

  • aws_session_token (str, optional) โ€“ The session token. Must be specified if not available in a config file or environment variables. Defaults to None.

  • client_config (dict, optional) โ€“ Kwargs dictionary for botocore.config.Config. Defaults to None.

  • transfer_config (dict, optional) โ€“ Kwargs dictionary for boto3.s3.transfer.TransferConfig. Defaults to None.