Implements a BERT wrapper around a ComposerTransformer.



BERT model based on ๐Ÿค— Transformers.

class composer.models.bert.model.BERTModel(module, config, tokenizer=None)[source]#

Bases: composer.models.transformer_shared.ComposerTransformer

BERT model based on ๐Ÿค— Transformers.

For more information, see Transformers.

  • module (BertModel) โ€“ An instance of BertModel that contains the forward pass function.

  • config (BertConfig) โ€“ The BertConfig object that stores information about the model hyperparameters.

  • tokenizer (BertTokenizer) โ€“ An instance of BertTokenizer. Necessary to process model inputs.

To create a BERT model for Language Model pretraining:

from composer.models import BERTModel
import transformers

config = transformers.BertConfig()
hf_model = transformers.BertLMHeadModel(config=config)
tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-uncased")
model = BERTModel(module=hf_model, config=config, tokenizer=tokenizer)

Runs the validation step.


batch (Dict) โ€“ a dictionary of Dict[str, Tensor] of inputs that the model expects, as found in ComposerTransformer.get_model_inputs().


tuple (Tensor, Tensor) โ€“ with the output from the forward pass and the correct labels. This is fed into directly into the output of ComposerModel.metrics().