composer.utils.ObjectStoreTransientError

exception composer.utils.ObjectStoreTransientError

Custom exception class to signify transient errors.

Implementations of the ObjectStore should re-raise any transient exceptions (e.g. too many requests, temporarily unavailable) with this class, so callers can easily detect whether they should attempt to retry any operation.

For example, the S3ObjectStore does the following:

from composer.utils import ObjectStore, ObjectStoreTransientError
import botocore.exceptions

class S3ObjectStore(ObjectStore):

    def upload_object(self, file_path: str, object_name: str):
        try:
            ...
        except botocore.exceptions.ClientError as e:
            if e.response['Error']['Code'] == 'LimitExceededException':
                raise ObjectStoreTransientError(e.response['Error']['Code']) from e
            raise e

Then, callers can automatically handle exceptions:

import time
from composer.utils import ObjectStore, ObjectStoreTransientError

def upload_file(object_store: ObjectStore, max_num_attempts: int = 3):
    for i in range(max_num_attempts):
        try:
            object_store.upload_object(...)
        except ObjectStoreTransientError:
            if i + 1 == max_num_attempts:
                raise
            else:
                # Try again after exponential back-off
                time.sleep(2**i)
        else:
            # upload successful
            return