composer.models.gpt2.model#
GPT-2 model based on Hugging Face GPT-2.
Implemented as a wrapper using ComposerTrainer
.
Classes
Implements |
- class composer.models.gpt2.model.GPT2Model(module, config, tokenizer=None, gradient_checkpointing=False)[source]#
Bases:
composer.models.transformer_shared.ComposerTransformer
Implements
ComposerTransformer
to wrap Hugging Face GPT-2 transformers. Logs training and validation perplexity.From Language Models are Unsupervised Multitask Learners (Radford et al, 2018).
- Parameters
module (transformers.GPT2Model) โ The model to wrap with this module.
config (transformers.GPT2Config) โ The config for the model.
tokenizer (transformers.GPT2Tokenizer) โ The tokenizer used for this model. Necessary to process model inputs.
gradient_checkpointing (bool, optional) โ Use gradient checkpointing. default:
False
.
To create a GPT-2 model for language modeling pretraining:
from composer.models import GPT2Model import transformers config = transformers.GPT2Config() hf_model = transformers.GPT2LMHeadModel(config=config) # gpt2-small model from huggingface tokenizer = transformers.GPT2Tokenizer.from_pretrained("gpt2") model = GPT2Model(module=hf_model, config=config, tokenizer=tokenizer)