streaming.base.storage

Base module for downloading/uploading files from/to cloud storage.

Functions

download_file	Use the correct download handler to download the file.
download_from_alipan	Download a file from remote Alipan to local.
download_from_azure	Download a file from remote Microsoft Azure to local.
download_from_azure_datalake	Download a file from remote Microsoft Azure Data Lake to local.
download_from_databricks_unity_catalog	Download a file from remote Databricks Unity Catalog to local.
download_from_dbfs	Download a file from remote Databricks File System to local.
download_from_gcs	Download a file from remote GCS to local.
download_from_hf	Download a file from remote Hugging Face to local.
download_from_local	Download a file from remote to local.
download_from_oci	Download a file from remote OCI to local.
download_from_s3	Download a file from remote AWS S3 (or any S3 compatible object store) to local.
download_from_sftp	Download a file from remote SFTP server to local filepath.

Classes

AzureDataLakeUploader	Upload file from local machine to Microsoft Azure DataLake.
AzureUploader	Upload file from local machine to Microsoft Azure bucket.
CloudUploader	Upload local files to a cloud storage.
GCSUploader	Upload file from local machine to Google Cloud Storage bucket.
HFUploader	Upload file from local machine to a Huggingface Dataset.
LocalUploader	Copy file from one local directory to another local directory.
OCIUploader	Upload file from local machine to Oracle Cloud Infrastructure (OCI) Cloud Storage.
S3Uploader	Upload file from local machine to AWS S3 bucket (or any S3 compatible object store).