🃏 Methods Overview# Alibi NLP Replace attention with AliBi AugMix CV Image-preserving data augmentations BlurPool CV Applies blur before pooling or downsampling ChannelsLast CV Uses channels last memory format (NHWC) ColOut CV Removes columns and rows from the image for augmentation and efficiency. CutMix CV Combines pairs of examples in non-overlapping regions and mixes labels CutOut CV Randomly erases rectangular blocks from the image. EMA CV NLP Maintains an exponential moving average of model weights for use in evaluation. Factorize CV NLP Factorize GEMMs into smaller GEMMs FusedLayerNorm NLP Fuses underlying LayerNorm kernels into single kernel GatedLinearUnits NLP Swaps the building block from a Linear layer to a Gated Linear layer. GhostBatchNorm CV Use smaller # samples to compute batchnorm GradientClipping CV NLP Clips all gradients in model based on specified clipping_type GyroDropout CV Clips all gradients in model based on specified clipping_type LabelSmoothing CV Smooths the labels with a uniform prior LayerFreezing CV NLP Progressively freezes layers during training. LowPrecisionGroupNorm CV NLP Forces GroupNorm to run in lower precision. LowPrecisionLayerNorm NLP CV Forces LayerNorm to run in lower precision. MixUp CV Blends pairs of examples and labels ProgressiveResizing CV Increases the input image size during training RandAugment CV Applies a series of random augmentations SAM CV SAM optimizer measures sharpness of optimization space SelectiveBackprop CV Drops examples with small loss contributions. SeqLengthWarmup NLP Progressively increase sequence length. SqueezeExcite CV Replaces eligible layers with Squeeze-Excite layers StochasticDepth CV Replaces a specified layer with a stochastic verion that randomly drops the layer or samples during training SWA CV NLP Computes running average of model weights. Weight Standardization CV Makes convolution weights always have zero mean and unit variance.