ActivationMonitor#

class composer.callbacks.ActivationMonitor(interval='25ba', ignore_module_types=None, only_log_wandb=True)[source]#

Logs stats of activation inputs and outputs.

This callback triggers at a user defined interval, and logs some simple statistics of the inputs, outputs for every torch module. This is done by attaching a forward hook to the module. Additionally, when after we finish logging we detach the forwards hook.

Example

>>> from composer import Trainer
>>> from composer.callbacks import ActivationMonitor
>>> # constructing trainer object with this callback
>>> trainer = Trainer(
...     model=model,
...     train_dataloader=train_dataloader,
...     eval_dataloader=eval_dataloader,
...     optimizers=optimizer,
...     max_duration="1ep",
...     callbacks=[ActivationMonitor()],
... )

The metrics are logged by the Logger to the following keys described below. Over an input of shape (batch, hid_dim), (batch, seq_len, hid_dim), etc. we compute statistics across hid_dim then take the average of these statistics. For convenience we have included example metrics logged:

Key

Logged data

activations/max/MODULE_NAME/input_{n}

The average max value of the hid_dim of the nth input activations into the current module.

activations/average/MODULE_NAME/input_{n}

The average value of the hid_dim input activations into the current module.

activations/l2_norm/MODULE_NAME/input_{n}

The average L2 Norm of the hid_dim of the nth input activations into the current module.

activations/kurtosis/MODULE_NAME/input_{n}

The average kurtosis of the hid_dim of the nth input activations into the current module.

activations/max/MODULE_NAME/output_{n}

The average max value of the hid_dim of the nth ouput activations of the current module.

activations/average/MODULE_NAME/output_{n}

The average value of the hid_dim of the output activations of the current module.

activations/l2_norm/MODULE_NAME/input_{n}

The average L2 Norm of the values of the hid_dim activations of the current module.

activations/kurtosis/MODULE_NAME/input_{n}

The average kurtosis of the hid_dim of the nth output activations of the current module.

Parameters
  • interval (Union[int, str, Time], optional) โ€“ Time string specifying how often to attach the logger and log the activations. For example, interval='5ba' means every 5 batches we log the activations. Default: โ€˜25baโ€™.

  • ignore_module_types (Optional[list[str]], optional) โ€“ A list of strings representing the class attributes we should ignore. For example passing in the list [โ€˜dropoutโ€™, โ€˜lnโ€™] will cause the class attributes that contain โ€˜dropoutโ€™ or โ€˜lnโ€™ to not be logged. Default: โ€˜Noneโ€™.

  • only_log_wandb (bool, optional) โ€“ A bool that determines if we should only log to Weights and Biases. This is recommended in partcular for larger models as this callback logs a lot. Default: โ€˜Trueโ€™.