ActivationMonitor

class composer.callbacks.ActivationMonitor(interval='25ba', ignore_module_types=None, only_log_wandb=True)

Logs stats of activation inputs and outputs.

This callback triggers at a user defined interval, and logs some simple statistics of the inputs, outputs for every torch module. This is done by attaching a forward hook to the module. Additionally, when after we finish logging we detach the forwards hook.

Example

>>> from composer import Trainer
>>> from composer.callbacks import ActivationMonitor
>>> # constructing trainer object with this callback
>>> trainer = Trainer(
...     model=model,
...     train_dataloader=train_dataloader,
...     eval_dataloader=eval_dataloader,
...     optimizers=optimizer,
...     max_duration="1ep",
...     callbacks=[ActivationMonitor()],
... )

The metrics are logged by the Logger to the following keys described below. Over an input of shape (batch, hid_dim), (batch, seq_len, hid_dim), etc. we compute statistics across hid_dim then take the average of these statistics. For convenience we have included example metrics logged:

Key	Logged data
activations/max/MODULE_NAME/input_{n}	The average max value of the hid_dim of the nth input activations into the current module.
activations/average/MODULE_NAME/input_{n}	The average value of the hid_dim input activations into the current module.
activations/l2_norm/MODULE_NAME/input_{n}	The average L2 Norm of the hid_dim of the nth input activations into the current module.
activations/kurtosis/MODULE_NAME/input_{n}	The average kurtosis of the hid_dim of the nth input activations into the current module.
activations/max/MODULE_NAME/output_{n}	The average max value of the hid_dim of the nth ouput activations of the current module.
activations/average/MODULE_NAME/output_{n}	The average value of the hid_dim of the output activations of the current module.
activations/l2_norm/MODULE_NAME/input_{n}	The average L2 Norm of the values of the hid_dim activations of the current module.
activations/kurtosis/MODULE_NAME/input_{n}	The average kurtosis of the hid_dim of the nth output activations of the current module.

Parameters

• interval (Union[int, str, Time], optional) – Time string specifying how often to attach the logger and log the activations. For example, interval='5ba' means every 5 batches we log the activations. Default: ‘25ba’.

• ignore_module_types (Optional[List[str]], optional) – A list of strings representing the class attributes we should ignore. For example passing in the list [‘dropout’, ‘ln’] will cause the class attributes that contain ‘dropout’ or ‘ln’ to not be logged. Default: ‘None’.

• only_log_wandb (bool, optional) – A bool that determines if we should only log to Weights and Biases. This is recommended in partcular for larger models as this callback logs a lot. Default: ‘True’.