all_gather

composer.utils.dist.all_gather(tensor)

Collects a Tensor from each rank.

See also

torch.distributed.all_gather()

Parameters

tensor (Tensor) – Tensor from each rank to be gathered.

Returns

Sequence[Tensor] – A sequence of tensors indexed by rank.