all_reduce#

composer.utils.dist.all_reduce(tensor, reduce_operation='SUM', group=None)[source]#

Reduce a tensor by applying the reduce_operation.

All ranks get the same, bitwise-identical result.

See also

torch.distributed.all_reduce()

Parameters

  • tensor (Tensor) โ€“ Input and output of the collective. The function operates in-place.

  • op (optional) โ€“ One of the values from torch.distributed.ReduceOp enum. Specifies an operation used for element-wise reductions.

  • group (ProcessGroup, optional) โ€“ The process group to work on. If None, the default process group will be used. Default is None.

  • tensor โ€“ Tensor to reduce. The function operates in-place.

  • reduce_operation (str, optional) โ€“

    The reduction operation (default: SUM).

    Valid options are:

    • SUM

    • PRODUCT

    • MIN

    • MAX

    • BAND

    • BOR

    • BXOR

Returns

None โ€“ tensor is modified in-place.