GyroDropout

class composer.algorithms.GyroDropout(p=0.5, sigma=256, tau=16)

Replaces all instances of torch.nn.Dropout with a GyroDropout.

By masking Dropout layer, this usually improves accuracy.

Parameters

• p (float, optional) – Float number of ratio to dropout. Default: 0.5.

• sigma (int, optional) – the number of total pre-selected subnetwork Default: 256.

• tau (int, optional) – the number of concurrently scheduled subnetworks in an iteration Default: 16.

Example

from composer.algorithms import GyroDropout

algorithm = GyroDropout(p=0.5, sigma=256, tau=16)
trainer = Trainer(
    model=model,
    train_dataloader=train_dataloader,
    max_duration="100ep",
    algorithms=[algorithm],
    optimizers=[optimizer]
)