CloudDownloader#

class streaming.base.storage.CloudDownloader[source]#

Download files from remote storage to a local filesystem.

abstract clean_up()[source]#: Clean up the downloader when it is done being used.

classmethod direct_download(remote, local, timeout=60.0)[source]#

Directly download a file from remote storage to local filesystem.

Parameters

remote (str | None) – Remote path.
local (str) – Local path.
timeout (float) – How long to wait for file to download before raising an exception. Defaults to 60 seconds.

Raises

ValueError – If the remote path is not provided while local does not exist or remote path is not supported.

download(remote, local, timeout=60.0)[source]#

Download a file from remote storage to local filesystem.

Parameters

remote (str | None) – Remote path.
local (str) – Local path.
timeout (float) – How long to wait for file to download before raising an exception. Defaults to 60 seconds.

Raises

ValueError – If the remote path does not contain the expected prefix or remote is not provided while local does not exist.

classmethod get(remote_dir=None)[source]#

Get the downloader for the remote path.