🃏 Methods Overview

Alibi

NLP

Replace attention with AliBi

AugMix

CV

Image-preserving data augmentations

BlurPool

CV

Applies blur before pooling or downsampling

ChannelsLast

CV

Uses channels last memory format (NHWC)

ColOut

CV

Removes columns and rows from the image for augmentation and efficiency.

CutMix

CV

Combines pairs of examples in non-overlapping regions and mixes labels

CutOut

CV

Randomly erases rectangular blocks from the image.

EMA

CV NLP

Maintains an exponential moving average of model weights for use in evaluation.

Factorize

CV NLP

Factorize GEMMs into smaller GEMMs

GatedLinearUnits

NLP

Swaps the building block from a Linear layer to a Gated Linear layer.

GhostBatchNorm

CV

Use smaller # samples to compute batchnorm

GradientClipping

CV NLP

Clips all gradients in model based on specified clipping_type

GyroDropout

CV

Clips all gradients in model based on specified clipping_type

LabelSmoothing

CV

Smooths the labels with a uniform prior

LayerFreezing

CV NLP

Progressively freezes layers during training.

LowPrecisionGroupNorm

CV NLP

Forces GroupNorm to run in lower precision.

LowPrecisionLayerNorm

NLP CV

Forces LayerNorm to run in lower precision.

MixUp

CV

Blends pairs of examples and labels

ProgressiveResizing

CV

Increases the input image size during training

RandAugment

CV

Applies a series of random augmentations

SAM

CV

SAM optimizer measures sharpness of optimization space

SelectiveBackprop

CV

Drops examples with small loss contributions.

SeqLengthWarmup

NLP

Progressively increase sequence length.

SqueezeExcite

CV

Replaces eligible layers with Squeeze-Excite layers

StochasticDepth

CV

Replaces a specified layer with a stochastic verion that randomly drops the layer or samples during training

SWA

CV NLP

Computes running average of model weights.

Weight Standardization

CV

Makes convolution weights always have zero mean and unit variance.