- class composer.DataSpec(dataloader, num_samples=None, num_tokens=None, device_transforms=None, split_batch=None, get_num_samples_in_batch=None, get_num_tokens_in_batch=None)#
Specifications for operating and training on data.
>>> # In this case, we apply NormalizationFn >>> # Construct DataSpec as shown below to apply this transformation >>> from composer.datasets.utils import NormalizationFn >>> CHANNEL_MEAN = (0.485 * 255, 0.456 * 255, 0.406 * 255) >>> CHANNEL_STD = (0.229 * 255, 0.224 * 255, 0.225 * 255) >>> device_transform_fn = NormalizationFn(mean=CHANNEL_MEAN, std=CHANNEL_STD) >>> train_dspec = DataSpec(train_dataloader, device_transforms=device_transform_fn) >>> # The same function can be used for eval dataloader as well >>> eval_dspec = DataSpec(eval_dataloader, device_transforms=device_transform_fn) >>> # Use this DataSpec object to construct trainer >>> trainer = Trainer( ... model=model, ... train_dataloader=train_dspec, ... eval_dataloader=eval_dspec, ... optimizers=optimizer, ... max_duration="1ep", ... )
dataloader (Union[Iterable, DataLoader]) – The dataloader, which can be any iterable that yields batches.
num_samples (int, optional) – The total number of samples in an epoch, across all ranks. This field is used by the
Timestamp(training progress tracker). If not specified, then
len(dataloader.dataset)is used (if this property is available). Otherwise, the dataset is assumed to be unsized.
device_transforms ((Batch) -> Batch, optional) – Function called by the
Trainerto modify the batch once it has been moved onto the device. For example, this function can be used for GPU-based normalization. It can modify the batch in-place, and it should return the modified batch. If not specified, the batch is not modified.
split_batch ((Batch, int) -> Sequence[Batch], optional) – Function called by the
Trainerto split a batch (the first parameter) into microbatches of a given size (the second parameter). If the
dataloaderyields batches not of type
torch.Tensor, Mapping, Tuple, or List, then this function must be specified.
get_num_samples_in_batch ((Batch) -> int, optional) –
Function that is called by the
Trainerto get the number of samples in the provided batch.
By default, if the batch contains tensors that all have the same 0th dim, then the value of the 0th dim will be returned. If the batch contains tensors where the 0th dim differ, then this function must be specified.
get_num_tokens_in_batch ((Batch) -> int, optional) –
Function that is called by the
Trainerto get the number of tokens in the provided batch.
By default, it checks for HuggingFace-style dictionary batches with
input_ids, and then checks
dataset.max_seq_len, and returns 0 if both of those fail, meaning that number of tokens processed will not be tracked as a part of the training progress tracking. Note that the defaults do NOT take padding into account, so if you want the token calculation to exclude padding, you should specify this function. This function must be specified to track the number of tokens processed during training in a non-default way.