LowPrecisionGroupNorm#

class composer.algorithms.LowPrecisionGroupNorm(apply_at=Event.INIT)[source]#

Replaces all instances of torch.nn.GroupNorm with class:.LPGroupNorm.

LPGroupNorm is a thin wrapper around torch.nn.GroupNorm which forces the layer to run in lower precision (torch.float16 or torch.bfloat16) if autocast is enabled. This algorithm has no effect in FP32 or DeepSpeed FP16 mode, where autocast is disabled.

This algorithm is intended to be used instead of Fused GroupNorm. They have similar behavior and performance.

Parameters: apply_at (Event) – Event where algorithm is applied.