apply_gradient_clipping#

composer.functional.apply_gradient_clipping(model, clipping_type, clipping_threshold, fsdp_enabled)[source]#

Clips all gradients in model based on specified clipping_type.

Parameters

  • model (ComposerModel or Module) โ€“ The model that we want to apply gradient clipping.

  • clipping_type ('adaptive', 'norm', 'value') โ€“ String denoting which type of gradient clipping to do. The options are: โ€˜normโ€™, which clips the gradient norm and uses torch.nn.utils.clip_grad_norm_, โ€˜valueโ€™, which clips gradient at a specified value and uses torch.nn.utils.clip_grad_value_, and โ€˜adaptiveโ€™, which clips all gradients based on gradient norm:parameter norm ratio using composer.algorithms.gradient_clipping.gradient_clipping._apply_agc.

  • clipping_threshold (float, optional) โ€“ Specifies what value to clip the gradients to (for โ€˜valueโ€™), what values to clip the gradient norms to (for โ€˜normโ€™), and threshold by which if grad_norm / weight_norm is greater than this threshold then scale gradients by this threshold * (weight_norm / grad_norm) (for โ€˜adaptiveโ€™).

  • fsdp_enabled (bool) โ€“ Bool of if the model is a FSDP model or not.