composer.utils#

Helper utilities.

Functions

add_vision_dataset_transform

Add a transform to a dataset's collection of transforms.

batch_get

Indexes into the batch given the key.

batch_set

Indexes into the batch given the key and sets the element at that index to value.

build_remote_backend

Build object store given the backend name and kwargs.

configure_excepthook

Collect and print system information when sys.excepthook() is called.

convert_flat_dict_to_nested_dict

Converts flat dictionary separated by slashes to nested dictionary.

convert_nested_dict_to_flat_dict

Takes in a nested dict converts it to a flat dict with keys separated by slashes.

create_fsdp_config

Modify fsdp_config to set default values for missing keys.

create_interval_scheduler

Helper function to create a scheduler according to a specified interval.

create_symlink_file

Create a symlink file, which can be followed by get_file().

disable_env_report

Disable environment report generation on exception.

enable_env_report

Enable environment report generation on exception.

ensure_folder_has_no_conflicting_files

Ensure that the given folder does not have any files conflicting with the filename format string.

ensure_folder_is_empty

Ensure that the given folder is empty.

ensure_tuple

Converts x into a tuple.

export_for_inference

Export a model for inference.

export_with_logger

Helper method for exporting a model for inference.

extract_hparams

Takes in local symbol table and recursively grabs any hyperparameter.

extract_path_from_symlink

Returns the checkpont path from symlink file.

format_name_with_dist

Format format_str with the run_name, distributed variables, and extra_format_kwargs.

format_name_with_dist_and_time

Format format_str with the run_name, distributed variables, timestamp, and extra_format_kwargs.

get_composer_env_dict

Query Composer pertinent system information as a dict.

get_compressor

Obtain the compressor that supports the format of the given file.

get_device

Takes string or Device and returns the corresponding Device.

get_file

Get a file from a local folder, URL, or object store.

get_free_tcp_port

Get free socket port to use as MASTER_PORT.

get_save_filename

Gets full filename of save filename.

import_object

Dynamically import a Python object (e.g.

is_compressed_pt

Whether the filename is for a directly compressed pt file.

is_hpu_installed

Determines whether the module needed for training on HPUs (Gaudi, Gaudi2) is installed.

is_model_deepspeed

Whether model is an instance of a DeepSpeedEngine.

is_model_fsdp

Whether model is an instance of a FullyShardedDataParallel.

is_notebook

Whether Composer is running in a IPython/Jupyter Notebook.

is_tar

Returns whether name has a tar-like extension.

is_xla_installed

Determines whether the module needed for training on TPUsโ€”torch_xlaโ€”is installed.

load_checkpoint

Load a checkpoint from a local file, URI, or cloud object store into state.

map_collection

Applies map_fn on each element in collection.

maybe_create_object_store_from_uri

Automatically creates an composer.utils.ObjectStore from supported URI formats.

maybe_create_remote_uploader_downloader_from_uri

Automatically creates a composer.loggers.RemoteUploaderDownloader from supported URI formats.

model_eval_mode

Set model.eval() for context duration, restoring model status at end.

parse_uri

Uses urllib.parse.urlparse() to parse the provided URI.

partial_format

Format a string with a partial set of arguments.

print_env

Generate system information report.

retry

Decorator to retry a function with backoff and jitter.

safe_torch_load

Load a torch checkpoint, catching errors due to backwards compatibility issues.

save_checkpoint

Checkpoint the training state.

validate_credentials

Upload a tiny text file to test if the credentials are setup correctly.

Classes

CliCompressor

Base class for data compression CLI tools.

EvalClient

Abstract class for implementing eval clients, such as LambdaEvalClient.

ExportFormat

Enum class for the supported export formats.

FSDPConfig

Configuration for Fully Sharded Data Parallelism (FSDP).

GCSObjectStore

Utility for uploading to and downloading from a Google Cloud bucket using google cloud storage sdk with either HMAC or service account authentications.

IteratorFileStream

Class used to convert iterator of bytes into a file-like binary stream object.

LambdaEvalClient

Utility for creating a client for and invoking an AWS Lambda.

LibcloudObjectStore

Utility for uploading to and downloading from object (blob) stores, such as Amazon S3.

LocalEvalClient

Utility for creating a client for and invoking local evaluations.

MLFlowObjectStore

Utility class for uploading and downloading artifacts from MLflow.

OCIObjectStore

Utility for uploading to and downloading from an OCI bucket.

ObjectStore

Abstract class for implementing object stores, such as LibcloudObjectStore and S3ObjectStore.

ParallelismConfig

Configuration for parallelism.

ParallelismType

Enum class for different parallelism types in the device mesh.

PartialFilePath

RemoteFilesExistingCheckStatus

An enumeration.

RemoteUploader

Class for uploading a file to object store asynchronously.

S3ObjectStore

Utility for uploading to and downloading from an S3-compatible bucket using boto3.

SFTPObjectStore

Utility for uploading to and downloading to a server via SFTP.

StringEnum

Base class for Enums containing string values.

TPConfig

Configuration for tensor parallelism (TP).

UCObjectStore

Utility class for uploading and downloading data from Databricks Unity Catalog (UC) Volumes.

Exceptions

MissingConditionalImportError

Handles errors for external packages that might not be installed.

ObjectStoreTransientError

Custom exception class to signify transient errors.

VersionedDeprecationWarning

A custom deprecation warning class that includes version information.