Timestamp#

class composer.Timestamp(epoch=0, batch=0, sample=0, token=0, batch_in_epoch=0, sample_in_epoch=0, token_in_epoch=0, total_wct=None, epoch_wct=None, batch_wct=None)[source]#

Timestamp represents a snapshot of the current training progress.

The timestamp measures training progress in terms of epochs, batches, samples, tokens, and wall clock time. Timestamps are not updated in-place.

See the Time Guide for more details on tracking time during training.

Parameters

epoch (int | Time[int], optional) – The epoch.
batch (int | Time[int], optional) – the batch.
sample (int | Time[int], optional) – The sample.
token (int | Time[int], optional) – The token.
batch_in_epoch (int | Time[int], optional) – The batch in the epoch.
sample_in_epoch (int | Time[int], optional) – The sample in the epoch.
token_in_epoch (int | Time[int], optional) – The token in the epoch.
total_wct (timedelta, optional) – The total wall-clock duration.
epoch_wct (timedelta, optional) – The wall-clock duration of the last epoch.
batch_wct (timedelta, optional) – The wall-clock duration of the last batch.

property batch#: The total batch count.

property batch_in_epoch#: The batch count in the current epoch (resets at 0 at the beginning of every epoch).

property batch_wct#: The wall-clock duration (in seconds) for the last batch.

copy(epoch=None, batch=None, sample=None, token=None, batch_in_epoch=None, sample_in_epoch=None, token_in_epoch=None, total_wct=None, epoch_wct=None, batch_wct=None)[source]#

Create a copy of the timestamp.

Any specified values will override the existing values in the returned copy.

Parameters

epoch (int | Time[int], optional) – The epoch.
batch (int | Time[int], optional) – the batch.
sample (int | Time[int], optional) – The sample.
token (int | Time[int], optional) – The token.
batch_in_epoch (int | Time[int], optional) – The batch in the epoch.
sample_in_epoch (int | Time[int], optional) – The sample in the epoch.
token_in_epoch (int | Time[int], optional) – The token in the epoch.
total_wct (timedelta, optional) – The elapsed duration from the beginning of training.

Returns

Timestamp – A new timestamp instance, created from a copy, but with any specified values overriding the existing values.

property epoch#: The total epoch count.

property epoch_wct#: The wall-clock duration (in seconds) for the current epoch.

get(unit)[source]#

Returns the current time in the specified unit.

Parameters: unit (str | TimeUnit) – The desired unit.
Returns: Time – The current time, in the specified unit.

get_state()[source]#

Returns all values of the timestamp object in a dictionary.

Returns: Dict[str, Union[Time[int], datetime.timedelta]] – All values of the timestamp object.

property sample#: The total sample count.

property sample_in_epoch#: The sample count in the current epoch (resets at 0 at the beginning of every epoch).

to_next_batch(samples=0, tokens=0, duration=None)[source]#

Create a new Timestamp, advanced to the next batch.

Equivalent to:

>>> timestamp.copy(
...     batch=timestamp.batch + 1,
...     batch_in_epoch=timestamp.batch_in_epoch + 1,
...     sample=timestamp.sample + samples,
...     sample_in_epoch=timestamp.sample_in_epoch + samples,
...     token = timestamp.token + tokens,
...     token_in_epoch=timestamp.token_in_epoch + tokens,
...     total_wct=timestamp.total_wct + duration,
...     epoch_wct=timestamp.epoch_wct + duration,
...     batch_wct=duration,
... )
Timestamp(...)

Note

For accurate time tracking, when doing distributed training, the samples and tokens should be the total across all ranks for the given batch. This method will not accumulate these counts automatically. If per-rank sample and token counts are provided, these counts will differ across ranks, which could lead towards inconsistent behavior by Algorithm or Callback instances that use these counts.

Parameters

samples (int | Time, optional) – The number of samples trained in the batch. Defaults to 0.
tokens (int | Time, optional) – The number of tokens trained in the batch. Defaults to 0.
duration (timedelta, optional) – The duration to train the batch.

to_next_epoch()[source]#

Create a new Timestamp, advanced to the next epoch.

Equivalent to:

>>> timestamp.copy(
...     epoch=timestamp.epoch+1,
...     batch_in_epoch=0,
...     sample_in_epoch=0,
...     token_in_epoch=0,
...     epoch_wct=datetime.timedelta(seconds=0),
...     batch_wct=datetime.timedelta(seconds=0),
... )
Timestamp(...)

property token#: The total token count.

property token_in_epoch#: The token count in the current epoch (resets at 0 at the beginning of every epoch).

property total_wct#: The wall-clock duration (in seconds) from the beginning of training.