RemoteUploader

class composer.utils.RemoteUploader(remote_folder, backend_kwargs=None, num_concurrent_uploads=2, num_attempts=3)

Class for uploading a file to object store asynchronously.

check_workers()

Non-blocking call to check workers are either running or done.

Traverse self.futures, and check if it’s completed 1. if it completed with exception, raise that exception 2. if it completed without exception, remove it from self.futures

upload_file_async(remote_file_name, file_path, overwrite)

Async call to submit a job for uploading.

It returns a future, so users can track the status of the individual future. User can also call wait() to wait for all the futures.

wait()

Blocking call to wait all the futures to complete.

If a future is done successfully, remove it from self.futures(), otherwise, raise the exception

wait_and_close()

Blocking call to wait all uploading to finish and close this uploader.

After this function is called, users can not use this uploader to uploading anymore. So please only call wait_and_close() after submitting all uploading requests.