composer.utils.dist.all_reduce(tensor, reduce_operation='SUM', group=None)[source]#

Reduce a tensor by applying the reduce_operation.

All ranks get the same, bitwise-identical result.

  • tensor (Tensor) โ€“ Input and output of the collective. The function operates in-place.

  • op (optional) โ€“ One of the values from torch.distributed.ReduceOp enum. Specifies an operation used for element-wise reductions.

  • group (ProcessGroup, optional) โ€“ The process group to work on. If None, the default process group will be used. Default is None.

  • tensor โ€“ Tensor to reduce. The function operates in-place.

  • reduce_operation (str, optional) โ€“

    The reduction operation (default: SUM).

    Valid options are:
    • SUM


    • MIN

    • MAX

    • BAND

    • BOR

    • BXOR


None โ€“ tensor is modified in-place.