GCSObjectStore

class composer.utils.GCSObjectStore(bucket, prefix='')

Utility for uploading to and downloading from a Google Cloud bucket using google cloud storage sdk with either HMAC or service account authentications. If both authentiations are available, this class will use service account authentication.

Warning

For best security practices, it is recommended to set credentials via environment variables or config files.

See guide to credentials for more information.

Parameters

• bucket (str) – The name of the Google Cloud bucket to upload to or download from.

• prefix (str, optional) – The prefix to use when uploading to or downloading from the bucket. Default is an empty string.

download_object(object_name, filename, overwrite=False, callback=None)

Downloads an object from the specified source in the cloud storage bucket and saves it to the given destination.

Parameters

• object_name (str) – The path to the object in the cloud storage bucket that needs to be downloaded.

• filename (Union[str, Path]) – The destination path where the object will be saved locally. It can be a string representing the file path or a pathlib.Path object.

• overwrite (bool, optional) – If set to True, the function will overwrite the destination file if it already exists. If set to False, and the destination file exists, a FileExistsError will be raised. Default is False.

• callback (Callable[[int, int], None], optional) – A callback function that can be used to track the progress of the download. It takes two integer arguments - the number of bytes downloaded and the total size of the object. Default is None. Unused for GCSObjectStore.

Raises

FileExistsError – If the destination file already exists and the overwrite parameter is set to False.

get_object_size(object_name)

Retrieves the size of an object stored in the cloud storage bucket.

Parameters

object_name (str) – The name of the object in the cloud storage bucket whose size is to be retrieved.

Returns

int – The size of the object in bytes.

Raises

• FileNotFoundError – If the specified object does not exist in the cloud storage bucket.

• Exception – If an error occurs while trying to retrieve the object’s size.

upload_object(object_name, filename, callback=None)

Uploads a file to the cloud storage bucket.

Parameters

• object_name (str, optional) – The destination path in the cloud storage bucket where the file will be saved. If not provided or an empty string is given, the file will be uploaded to the root of the bucket with the same name as the source file. Default is an empty string.

• filename (Union[str, Path]) – The path to the local file

• callback – optional