GradMonitor

class composer.callbacks.grad_monitor.GradMonitor(log_layer_grad_norms: bool = False)

Bases: composer.core.callback.Callback

Logs the L2 norm.

This callback logs the L2 norm of all the parameters in the model, and optionally, the L2 norm of the parameters by each layer.

It logs the L2 norm on each batch under the grad_l2_norm/step key. If log_layer_grad_norms is True (default False), then layer-wise L2 norms are logged under layer_grad_l2_norm/LAYER_NAME.

Parameters

log_layer_grad_norms (bool, optional) – Whether to log the L2 normalization of each layer. Defaults to False.

after_train_batch(state: State, logger: Logger)

Compute the gradient L2 norm after the reduction of the backwards pass across GPUs. This function iterates over the parameters of the model and hence may cause a reduction in throughput while training large models. In order to ensure correctness, this function should be called after gradient unscaling in cases where gradients are scaled.

Parameters

• state (State) – The State object used during training.

• logger (Logger) – The Logger object.