CosineAnnealingWarmRestartsScheduler#
- class composer.optim.CosineAnnealingWarmRestartsScheduler(t_0, t_mult=1.0, alpha_f=0.0)[source]#
Cyclically decays the learning rate according to the decreasing part of a cosine curve.
See also
This scheduler is based on
CosineAnnealingWarmRestarts
from PyTorch.This scheduler resembles a regular cosine annealing curve, as seen in
CosineAnnealingScheduler
, except that after the curve first completest_0
time, the curve resets to the start. The durations of subsequent cycles are each multiplied byt_mult
.Specifically, the learning rate multiplier can be expressed as:
Given , the fraction of time elapsed through the cycle, as:
Where represents the period of the first cycle, represents the multiplier for the duration of successive cycles, and represents the learning rate multiplier to decay to.