Contents Menu Expand Light mode Dark mode Auto light/dark mode
Composer
Light Logo Dark Logo
Star

Getting Started

  • ๐Ÿ’พ Installation
  • ๐Ÿš€ Quick Start
  • ๐ŸšŒ Welcome Tour

Tutorials

  • ๐Ÿ–ผ๏ธ Getting Started
  • ฦ’() Functional API
  • ๐Ÿฉบ Image Segmentation
  • ๐Ÿค– Custom Speedup Methods
  • ๐ŸŽ๏ธ FFCV DataLoaders
  • ๐Ÿค— Finetuning Hugging Face Models
  • ๐Ÿค— Pretraining and Finetuning with Hugging Face Models
  • โšก Migrating from PTL
  • ๐Ÿ›‘ Early Stopping
  • โ™ป๏ธ Auto Microbatching
  • โฏ๏ธ Autoresume Training
  • ๐Ÿฅก Exporting for Inference
  • ๐Ÿ”Œ Training with TPUs
  • ๐Ÿ•น๏ธ Distributed Training with Submitit
  • โ˜๏ธ Train ResNet-50 on AWS

Speedup Methods

  • ๐Ÿค– Algorithms
  • ฦ’() Functional

Trainer

  • โš™๏ธ Using the Trainer
  • ๐Ÿ›ป ComposerModel
  • ๐Ÿ’ฟ DataLoaders
  • ๐Ÿ“Š Evaluation
  • ๐Ÿ“‰ Schedulers
  • โŒ› Time
  • ๐Ÿ“† Events
  • โœ… Checkpointing
  • ๐Ÿชต Logging
  • ๐Ÿ›• File Uploading
  • โ˜Ž๏ธ Callbacks
  • ๐ŸŽ๏ธ Performance
    • โฑ๏ธ Performance Profiling
    • ๐Ÿฅฝ Analyzing Traces

Notes

  • ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Distributed Training
  • ๐Ÿ›‘ Early Stopping
  • ๐Ÿ”ข Numerics
  • โ™ป๏ธ Automatic Microbatching
  • โฏ Auto Resumption
  • ๐Ÿ“‹ Tensorboard Monitoring
  • ๐Ÿƒโ€โ™€๏ธ Run Name

Methods Library

  • ๐Ÿƒ Methods Overview
  • ๐Ÿฅธ ALiBi
  • ๐ŸŽจ AugMix
  • ๐ŸŠ BlurPool
  • ๐Ÿ“บ Channels Last
  • ๐Ÿ›๏ธ ColOut
  • โœ‚๏ธ CutMix
  • ๐ŸŽƒ Cutout
  • ๐Ÿ‹๏ธโ€โ™€๏ธ Decoupled Weight Decay
  • ๐Ÿšš EMA
  • โž— Factorize
  • ๐Ÿฐ Fused LayerNorm
  • โ›ฉ๏ธ Gated Linear Units
  • ๐Ÿ‘ป Ghost BatchNorm
  • ๐Ÿ“Ž Gradient Clipping
  • ๐Ÿฅ™ Gyro Dropout
  • ๐Ÿงˆ Label Smoothing
  • โ„๏ธ Layer Freezing
  • ๐ŸŽ‚ Low Precision GroupNorm
  • ๐Ÿง Low Precision LayerNorm
  • ๐Ÿฅฃ MixUp
  • ๐Ÿž๏ธ Progressive Image Resizing
  • ๐ŸŽฒ RandAugment
  • ๐Ÿ”๏ธ Sharpness Aware Minimization (SAM)
  • โš–๏ธ Scale Schedule
  • โฎ๏ธ Selective Backprop
  • ๐Ÿ”† Sequence Length Warmup
  • ๐Ÿซ€ Squeeze-and-Excitation
  • ๐ŸงŠ Stochastic Depth (Block)
  • ๐ŸŽฐ Stochastic Depth (Sample)
  • ๐Ÿงฉ Stochastic Weight Averaging
  • ๐„ท Weight Standardization

Model Library

  • ๐Ÿฆญ BERT
  • ๐Ÿ‘€ CIFAR ResNet
  • ๐Ÿคฟ DeepLabv3+
  • ๐Ÿ•ธ๏ธ EfficientNet
  • ๐Ÿ“š GPT-2
  • ๐Ÿ™๏ธ ResNet
  • โ†ฉ๏ธ UNet

API Reference

  • composer
    • Algorithm
    • Callback
    • ComposerModel
    • DataSpec
    • Engine
    • Evaluator
    • Event
    • Logger
    • State
    • Time
    • TimeUnit
    • Timestamp
    • Trainer
  • composer.algorithms
    • Alibi
    • AugMix
    • AugmentAndMixTransform
    • BlurPool
    • ChannelsLast
    • ColOut
    • ColOutTransform
    • CutMix
    • CutOut
    • EMA
    • Factorize
    • FusedLayerNorm
    • GatedLinearUnits
    • GhostBatchNorm
    • GradientClipping
    • GyroDropout
    • LabelSmoothing
    • LayerFreezing
    • LowPrecisionGroupNorm
    • LowPrecisionLayerNorm
    • MixUp
    • NoOpModel
    • ProgressiveResizing
    • RandAugment
    • RandAugmentTransform
    • SAM
    • SWA
    • SelectiveBackprop
    • SeqLengthWarmup
    • SqueezeExcite
    • SqueezeExcite2d
    • SqueezeExciteConv2d
    • StochasticDepth
    • WeightStandardization
  • composer.callbacks
    • ActivationMonitor
    • CheckpointSaver
    • EarlyStopper
    • ExportForInferenceCallback
    • HealthChecker
    • ImageVisualizer
    • LRMonitor
    • MLPerfCallback
    • MemoryMonitor
    • OptimizerMonitor
    • RuntimeEstimator
    • SpeedMonitor
    • ThresholdStopper
  • composer.core
    • ensure_data_spec
    • ensure_evaluator
    • ensure_time
    • get_precision_context
    • validate_eval_automicrobatching
    • MemoryFormat
    • Precision
    • PyTorchScheduler
    • Serializable
    • Trace
    • TrainerMode
    • composer.core.BreakEpochException
  • composer.core.types
    • MemoryFormat
    • PyTorchScheduler
    • TrainerMode
    • composer.core.types.BreakEpochException
  • composer.datasets
    • build_ade20k_dataloader
    • build_cifar10_dataloader
    • build_ffcv_cifar10_dataloader
    • build_ffcv_imagenet_dataloader
    • build_imagenet_dataloader
    • build_lm_dataloader
    • build_mnist_dataloader
    • build_streaming_ade20k_dataloader
    • build_streaming_c4_dataloader
    • build_streaming_cifar10_dataloader
    • build_streaming_imagenet1k_dataloader
    • build_synthetic_ade20k_dataloader
    • build_synthetic_cifar10_dataloader
    • build_synthetic_imagenet_dataloader
    • build_synthetic_mnist_dataloader
    • ADE20k
    • PytTrain
    • PytVal
    • SyntheticBatchPairDataset
    • SyntheticDataLabelType
    • SyntheticDataType
    • SyntheticPILDataset
  • composer.devices
    • Device
    • DeviceCPU
    • DeviceGPU
    • DeviceMPS
    • DeviceTPU
  • composer.functional
    • apply_alibi
    • apply_blurpool
    • apply_channels_last
    • apply_factorization
    • apply_fused_layernorm
    • apply_gated_linear_units
    • apply_ghost_batchnorm
    • apply_gradient_clipping
    • apply_gyro_dropout
    • apply_low_precision_groupnorm
    • apply_low_precision_layernorm
    • apply_squeeze_excite
    • apply_stochastic_depth
    • apply_weight_standardization
    • augmix_image
    • colout_batch
    • compute_ema
    • cutmix_batch
    • cutout_batch
    • freeze_layers
    • mixup_batch
    • randaugment_image
    • resize_batch
    • select_using_loss
    • set_batch_sequence_length
    • should_selective_backprop
    • smooth_labels
  • composer.loggers
    • CometMLLogger
    • ConsoleLogger
    • FileLogger
    • InMemoryLogger
    • LoggerDestination
    • MLFlowLogger
    • MosaicMLLogger
    • ProgressBarLogger
    • RemoteUploaderDownloader
    • SlackLogger
    • TensorboardLogger
    • WandBLogger
  • composer.loss
    • binary_cross_entropy_with_logits
    • soft_cross_entropy
    • DiceLoss
  • composer.metrics
    • BinaryF1Score
    • CrossEntropy
    • Dice
    • HFCrossEntropy
    • InContextLearningLMAccuracy
    • InContextLearningLMExpectedCalibrationError
    • InContextLearningMCExpectedCalibrationError
    • InContextLearningMetric
    • InContextLearningMultipleChoiceAccuracy
    • InContextLearningQAAccuracy
    • LanguageCrossEntropy
    • LanguagePerplexity
    • LossMetric
    • MAP
    • MIoU
    • MaskedAccuracy
    • Perplexity
  • composer.models
    • composer_deeplabv3
    • composer_efficientnetb0
    • composer_resnet
    • composer_resnet_cifar
    • composer_timm
    • create_bert_classification
    • create_bert_mlm
    • create_gpt2
    • mnist_model
    • vit_small_patch16
    • write_huggingface_pretrained_from_composer_checkpoint
    • ComposerClassifier
    • HuggingFaceModel
    • Initializer
    • MMDetModel
    • UNet
  • composer.optim
    • compile_composer_scheduler
    • ComposerScheduler
    • ConstantScheduler
    • ConstantWithWarmupScheduler
    • CosineAnnealingScheduler
    • CosineAnnealingWarmRestartsScheduler
    • CosineAnnealingWithWarmupScheduler
    • DecoupledAdamW
    • DecoupledSGDW
    • ExponentialScheduler
    • LinearScheduler
    • LinearWithWarmupScheduler
    • MultiStepScheduler
    • MultiStepWithWarmupScheduler
    • PolynomialScheduler
    • PolynomialWithWarmupScheduler
    • StepScheduler
  • composer.profiler
    • cyclic_schedule
    • JSONTraceHandler
    • Marker
    • Profiler
    • ProfilerAction
    • SystemProfiler
    • TorchProfiler
    • TraceHandler
  • composer.utils
    • batch_get
    • batch_set
    • configure_excepthook
    • convert_flat_dict_to_nested_dict
    • convert_nested_dict_to_flat_dict
    • create_symlink_file
    • disable_env_report
    • enable_env_report
    • ensure_folder_has_no_conflicting_files
    • ensure_folder_is_empty
    • ensure_tuple
    • export_for_inference
    • export_with_logger
    • extract_hparams
    • format_name_with_dist
    • format_name_with_dist_and_time
    • get_composer_env_dict
    • get_device
    • get_file
    • get_free_tcp_port
    • import_object
    • is_model_deepspeed
    • is_model_fsdp
    • is_notebook
    • is_tar
    • is_tpu_installed
    • load_checkpoint
    • map_collection
    • maybe_create_object_store_from_uri
    • maybe_create_remote_uploader_downloader_from_uri
    • model_eval_mode
    • parse_uri
    • print_env
    • retry
    • safe_torch_load
    • save_checkpoint
    • using_torch_2
    • ExportFormat
    • IteratorFileStream
    • LibcloudObjectStore
    • OCIObjectStore
    • ObjectStore
    • PartialFilePath
    • S3ObjectStore
    • SFTPObjectStore
    • StringEnum
    • composer.utils.MissingConditionalImportError
    • composer.utils.ObjectStoreTransientError
  • composer.utils.dist
    • all_gather
    • all_gather_object
    • all_reduce
    • barrier
    • broadcast
    • broadcast_object_list
    • get_global_rank
    • get_local_rank
    • get_local_world_size
    • get_node_rank
    • get_sampler
    • get_world_size
    • initialize_dist
    • is_available
    • is_initialized
  • composer.utils.reproducibility
    • configure_deterministic_mode
    • get_random_seed
    • get_rng_state
    • load_rng_state
    • seed_all
Back to top
Edit this page

barrier#

composer.utils.dist.barrier()[source]#

Synchronizes all processes.

This function blocks until all processes reach this function.

See also

torch.distributed.barrier()

Next
broadcast
Previous
all_reduce
Copyright © 2022, MosaicML, Inc.
Made with Sphinx and @pradyunsg's Furo