composer.utils#

Helper utilities.

Functions

`add_vision_dataset_transform`	Add a transform to a dataset's collection of transforms.
`batch_get`	Indexes into the batch given the key.
`batch_set`	Indexes into the batch given the key and sets the element at that index to value.
`build_remote_backend`	Build object store given the backend name and kwargs.
`configure_excepthook`	Collect and print system information when `sys.excepthook()` is called.
`convert_flat_dict_to_nested_dict`	Converts flat dictionary separated by slashes to nested dictionary.
`convert_nested_dict_to_flat_dict`	Takes in a nested dict converts it to a flat dict with keys separated by slashes.
`create_fsdp_config`	Modify fsdp_config to set default values for missing keys.
`create_interval_scheduler`	Helper function to create a scheduler according to a specified interval.
`create_symlink_file`	Create a symlink file, which can be followed by `get_file()`.
`disable_env_report`	Disable environment report generation on exception.
`enable_env_report`	Enable environment report generation on exception.
`ensure_folder_has_no_conflicting_files`	Ensure that the given folder does not have any files conflicting with the `filename` format string.
`ensure_folder_is_empty`	Ensure that the given folder is empty.
`ensure_tuple`	Converts `x` into a tuple.
`export_for_inference`	Export a model for inference.
`export_with_logger`	Helper method for exporting a model for inference.
`extract_hparams`	Takes in local symbol table and recursively grabs any hyperparameter.
`extract_path_from_symlink`	Returns the checkpont path from symlink file.
`format_name_with_dist`	Format `format_str` with the `run_name`, distributed variables, and `extra_format_kwargs`.
`format_name_with_dist_and_time`	Format `format_str` with the `run_name`, distributed variables, `timestamp`, and `extra_format_kwargs`.
`get_composer_env_dict`	Query Composer pertinent system information as a dict.
`get_compressor`	Obtain the compressor that supports the format of the given file.
`get_device`	Takes string or Device and returns the corresponding `Device`.
`get_file`	Get a file from a local folder, URL, or object store.
`get_free_tcp_port`	Get free socket port to use as MASTER_PORT.
`get_save_filename`	Gets full filename of save filename.
`import_object`	Dynamically import a Python object (e.g.
`is_compressed_pt`	Whether the filename is for a directly compressed pt file.
`is_hpu_installed`	Determines whether the module needed for training on HPUs (Gaudi, Gaudi2) is installed.
`is_model_deepspeed`	Whether `model` is an instance of a `DeepSpeedEngine`.
`is_model_fsdp`	Whether `model` is an instance of a `FullyShardedDataParallel`.
`is_notebook`	Whether Composer is running in a IPython/Jupyter Notebook.
`is_tar`	Returns whether `name` has a tar-like extension.
`is_xla_installed`	Determines whether the module needed for training on TPUs—torch_xla—is installed.
`load_checkpoint`	Load a checkpoint from a local file, URI, or cloud object store into `state`.
`map_collection`	Applies `map_fn` on each element in `collection`.
`maybe_create_object_store_from_uri`	Automatically creates an `composer.utils.ObjectStore` from supported URI formats.
`maybe_create_remote_uploader_downloader_from_uri`	Automatically creates a `composer.loggers.RemoteUploaderDownloader` from supported URI formats.
`model_eval_mode`	Set model.eval() for context duration, restoring model status at end.
`parse_uri`	Uses `urllib.parse.urlparse()` to parse the provided URI.
`partial_format`	Format a string with a partial set of arguments.
`print_env`	Generate system information report.
`retry`	Decorator to retry a function with backoff and jitter.
`safe_torch_load`	Load a torch checkpoint, catching errors due to backwards compatibility issues.
`save_checkpoint`	Checkpoint the training `state`.
`validate_credentials`	Upload a tiny text file to test if the credentials are setup correctly.

Classes

`CliCompressor`	Base class for data compression CLI tools.
`EvalClient`	Abstract class for implementing eval clients, such as LambdaEvalClient.
`ExportFormat`	Enum class for the supported export formats.
`FSDPConfig`	Configuration for Fully Sharded Data Parallelism (FSDP).
`GCSObjectStore`	Utility for uploading to and downloading from a Google Cloud bucket using `google cloud storage sdk` with either HMAC or service account authentications.
`IteratorFileStream`	Class used to convert iterator of bytes into a file-like binary stream object.
`LambdaEvalClient`	Utility for creating a client for and invoking an AWS Lambda.
`LibcloudObjectStore`	Utility for uploading to and downloading from object (blob) stores, such as Amazon S3.
`LocalEvalClient`	Utility for creating a client for and invoking local evaluations.
`MLFlowObjectStore`	Utility class for uploading and downloading artifacts from MLflow.
`OCIObjectStore`	Utility for uploading to and downloading from an OCI bucket.
`ObjectStore`	Abstract class for implementing object stores, such as LibcloudObjectStore and S3ObjectStore.
`ParallelismConfig`	Configuration for parallelism.
`ParallelismType`	Enum class for different parallelism types in the device mesh.
`PartialFilePath`
`RemoteFilesExistingCheckStatus`	An enumeration.
`RemoteUploader`	Class for uploading a file to object store asynchronously.
`S3ObjectStore`	Utility for uploading to and downloading from an S3-compatible bucket using `boto3`.
`SFTPObjectStore`	Utility for uploading to and downloading to a server via SFTP.
`StringEnum`	Base class for Enums containing string values.
`TPConfig`	Configuration for tensor parallelism (TP).
`UCObjectStore`	Utility class for uploading and downloading data from Databricks Unity Catalog (UC) Volumes.

Exceptions

`MissingConditionalImportError`	Handles errors for external packages that might not be installed.
`ObjectStoreTransientError`	Custom exception class to signify transient errors.
`VersionedDeprecationWarning`	A custom deprecation warning class that includes version information.