MemoryMonitor#
- class composer.callbacks.MemoryMonitor[source]#
Logs the memory usage of the model.
This callback calls the torch memory stats API for CUDA (see
torch.cuda.memory_stats()
) on theEvent.AFTER_TRAIN_BATCH
and reports different memory statistics.Example
>>> from composer import Trainer >>> from composer.callbacks import MemoryMonitor >>> # constructing trainer object with this callback >>> trainer = Trainer( ... model=model, ... train_dataloader=train_dataloader, ... eval_dataloader=eval_dataloader, ... optimizers=optimizer, ... max_duration="1ep", ... callbacks=[MemoryMonitor()], ... )
The memory statistics are logged by the
Logger
to the following keys as described below.Key
Logged data
memory/{statistic}
Several memory usage statistics are logged on
Event.AFTER_TRAIN_BATCH
event.The following statistics are recorded:
Statistic
Description
alloc_requests
Number of memory allocation requests received by the memory allocator.
free_requests
Number of memory free requests received by the memory allocator.
allocated_mem
Amount of allocated memory in bytes.
active_mem
Amount of active memory in bytes at the time of recording.
inactive_mem
Amount of inactive, non-releaseable memory in bytes at the time of recording.
reserved_mem
Amount of reserved memory in bytes at the time of recording.
alloc_retries
Number of failed cudaMalloc calls that result in a cache flush and retry.
Note
Memory usage monitoring is only supported for GPU devices.