- class streaming.base.storage.CloudUploader(out, keep_local=False, progress_bar=False, retry=2, exist_ok=False)#
Upload local files to a cloud storage.
Remove the local file if it is enabled.
local (str) – A local file path.
- classmethod get(out, keep_local=False, progress_bar=False, retry=2, exist_ok=False)#
Instantiate a cloud provider uploader or a local uploader based on remote path.
Output dataset directory to save shard files.
outis a local directory, shard files are saved locally.
outis a remote directory, a local temporary directory is created to cache the shard files and then the shard files are uploaded to a remote location. At the end, the temp directory is deleted once shards are uploaded.
outis a tuple of
(local_dir, remote_dir), shard files are saved in the local_dir and also uploaded to a remote location.
keep_local (bool) – If the dataset is uploaded, whether to keep the local dataset shard file or remove it after uploading. Defaults to
progress_bar (bool) – Display TQDM progress bars for uploading output dataset files to a remote location. Default to
retry (int) – Number of times to retry uploading a file. Defaults to
exist_ok (bool) – When exist_ok = False, raise error if the local part of
outalready exists and has contents. Defaults to
CloudUploader – An instance of sub-class.
List all objects in the object store with the given prefix.
prefix (Optional[str], optional) – The prefix to search for. Defaults to
List[str] – A list of object names that match the prefix.