composer.distributed#

Distributed training.

Functions

ddp_sync_context

A context manager for handling the DDPSyncStrategy.

prepare_ddp_module

Wraps the module in a torch.nn.parallel.DistributedDataParallel object if running distributed training.

prepare_fsdp_module

Prepare a module (assumed ComposerModel) and optimizer for use with torch.distributed.fsdp.FullyShardedDataParallel.

prepare_tp_module

Prepare a module (assumed ComposerModel) for use with tensor parallel.

Classes

DDPSyncStrategy

How and when gradient synchronization should happen.