composer.datasets.cifar

CIFAR image classification dataset.

The CIFAR datasets are a collection of labeled 32x32 colour images. Please refer to the CIFAR dataset for more details.

Hparams

These classes are used with yahp for YAML-based configuration.

CIFAR100WebDatasetHparams	Defines an instance of the CIFAR-100 WebDataset for image classification.
CIFAR10DatasetHparams	Defines an instance of the CIFAR-10 dataset for image classification from a local disk.
CIFAR10WebDatasetHparams	Defines an instance of the CIFAR-10 WebDataset for image classification.
CIFAR20WebDatasetHparams	Defines an instance of the CIFAR-20 WebDataset for image classification.
CIFARWebDatasetHparams	Common functionality for CIFAR WebDatasets.

class composer.datasets.cifar.CIFAR100WebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256, remote='s3://mosaicml-internal-dataset-cifar100', name='cifar100', n_train_samples=50000, n_val_samples=10000, height=32, width=32, n_classes=100, channel_means=(0.5071, 0.4867, 0.4408), channel_stds=(0.2675, 0.2565, 0.2761))

Bases: composer.datasets.cifar.CIFARWebDatasetHparams

Defines an instance of the CIFAR-100 WebDataset for image classification.

Parameters

• datadir (str) – The path to the data directory.

• is_train (bool) – Whether to load the training data or validation data. Default: True.

• drop_last (bool) – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle (bool) – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir (str) – WebDataset cache directory.

• webdataset_cache_verbose (str) – WebDataset cache verbosity.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir – WebDataset cache directory.

• webdataset_cache_verbose – WebDataset cache verbosity.

• remote (str) – S3 bucket or root directory where dataset is stored.

• name (str) – Key used to determine where dataset is cached on local filesystem.

• n_train_samples (int) – Number of training samples.

• n_val_samples (int) – Number of validation samples.

• height (int) – Sample image height in pixels. Default: 32.

• width (int) – Sample image width in pixels. Default: 32.

• n_classes (int) – Number of output classes.

• channel_means (list of float) – Channel means for normalization.

• channel_stds (list of float) – Channel stds for normalization.

• remote – S3 bucket or root directory where dataset is stored. Default: 's3://mosaicml-internal-dataset-cifar100'.

• name – Key used to determine where dataset is cached on local filesystem. Default: 'cifar100'.

• n_train_samples – Number of training samples. Default: 50000.

• n_val_samples – Number of validation samples. Default: 10000.

• n_classes – Number of output classes. Default: 100.

• channel_means – Channel means for normalization. Default: (0.5071, 0.4867, 0.4408).

• channel_stds – Channel stds for normalization. Default: (0.2675, 0.2565, 0.2761).

class composer.datasets.cifar.CIFAR10DatasetHparams(use_synthetic=False, synthetic_num_unique_samples=100, synthetic_device='cpu', synthetic_memory_format=MemoryFormat.CONTIGUOUS_FORMAT, is_train=True, drop_last=True, shuffle=True, datadir=None, download=True, use_ffcv=False, ffcv_dir='/tmp', ffcv_dest_train='cifar_train.ffcv', ffcv_dest_val='cifar_val.ffcv', ffcv_write_dataset=False)

Bases: composer.datasets.hparams.DatasetHparams, composer.datasets.hparams.SyntheticHparamsMixin

Defines an instance of the CIFAR-10 dataset for image classification from a local disk.

Parameters

• use_synthetic (bool, optional) – Whether to use synthetic data. Default: False.

• synthetic_num_unique_samples (int, optional) – The number of unique samples to allocate memory for. Ignored if use_synthetic is False. Default: 100.

• synthetic_device (str, optional) – The device to store the sample pool on. Set to 'cuda' to store samples on the GPU and eliminate PCI-e bandwidth with the dataloader. Set to 'cpu' to move data between host memory and the device on every batch. Ignored if use_synthetic is False. Default: 'cpu'.

• synthetic_memory_format – The MemoryFormat to use. Ignored if use_synthetic is False. Default: 'CONTIGUOUS_FORMAT'.

• datadir (str) – The path to the data directory.

• is_train (bool) – Whether to load the training data or validation data. Default: True.

• drop_last (bool) – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle (bool) – Whether to shuffle the dataset. Default: True.

• download (bool) – Whether to download the dataset, if needed. Default: True.

• use_ffcv (bool) – Whether to use FFCV dataloaders. Default: False.

• ffcv_dir (str) – A directory containing train/val <file>.ffcv files. If these files don’t exist and ffcv_write_dataset is True, train/val <file>.ffcv files will be created in this dir. Default: "/tmp".

• ffcv_dest_train (str) – <file>.ffcv file that has training samples. Default: "cifar_train.ffcv".

• ffcv_dest_val (str) – <file>.ffcv file that has validation samples. Default: "cifar_val.ffcv".

• ffcv_write_dataset (std) – Whether to create dataset in FFCV format (<file>.ffcv) if it doesn’t exist. Default:

• False. –

initialize_object(batch_size, dataloader_hparams)

Creates a DataLoader or DataSpec for this dataset.

Parameters

• batch_size (int) – The size of the batch the dataloader should yield. This batch size is device-specific and already incorporates the world size.

• dataloader_hparams (DataLoaderHparams) – The dataset-independent hparams for the dataloader.

Returns

DataLoader or DataSpec – The DataLoader, or if the dataloader yields batches of custom types, a DataSpec.

class composer.datasets.cifar.CIFAR10WebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256, remote='s3://mosaicml-internal-dataset-cifar10', name='cifar10', n_train_samples=50000, n_val_samples=10000, height=32, width=32, n_classes=10, channel_means=(0.4914, 0.4822, 0.4465), channel_stds=(0.247, 0.243, 0.261))

Bases: composer.datasets.cifar.CIFARWebDatasetHparams

Defines an instance of the CIFAR-10 WebDataset for image classification.

Parameters

• datadir (str) – The path to the data directory.

• is_train (bool) – Whether to load the training data or validation data. Default: True.

• drop_last (bool) – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle (bool) – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir (str) – WebDataset cache directory.

• webdataset_cache_verbose (str) – WebDataset cache verbosity.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir – WebDataset cache directory.

• webdataset_cache_verbose – WebDataset cache verbosity.

• remote (str) – S3 bucket or root directory where dataset is stored.

• name (str) – Key used to determine where dataset is cached on local filesystem.

• n_train_samples (int) – Number of training samples.

• n_val_samples (int) – Number of validation samples.

• height (int) – Sample image height in pixels. Default: 32.

• width (int) – Sample image width in pixels. Default: 32.

• n_classes (int) – Number of output classes.

• channel_means (list of float) – Channel means for normalization.

• channel_stds (list of float) – Channel stds for normalization.

• remote – S3 bucket or root directory where dataset is stored. Default: 's3://mosaicml-internal-dataset-cifar10'.

• name – Key used to determine where dataset is cached on local filesystem. Default: 'cifar10'.

• n_train_samples – Number of training samples. Default: 50000.

• n_val_samples – Number of validation samples. Default: 10000.

• n_classes – Number of output classes. Default: 10.

• channel_means – Channel means for normalization. Default: (0.4914, 0.4822, 0.4465).

• channel_stds – Channel stds for normalization. Default: (0.247, 0.243, 0.261).

class composer.datasets.cifar.CIFAR20WebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256, remote='s3://mosaicml-internal-dataset-cifar20', name='cifar20', n_train_samples=50000, n_val_samples=10000, height=32, width=32, n_classes=20, channel_means=(0.5071, 0.4867, 0.4408), channel_stds=(0.2675, 0.2565, 0.2761))

Bases: composer.datasets.cifar.CIFARWebDatasetHparams

Defines an instance of the CIFAR-20 WebDataset for image classification.

Parameters

• datadir (str) – The path to the data directory.

• is_train (bool) – Whether to load the training data or validation data. Default: True.

• drop_last (bool) – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle (bool) – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir (str) – WebDataset cache directory.

• webdataset_cache_verbose (str) – WebDataset cache verbosity.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir – WebDataset cache directory.

• webdataset_cache_verbose – WebDataset cache verbosity.

• remote (str) – S3 bucket or root directory where dataset is stored.

• name (str) – Key used to determine where dataset is cached on local filesystem.

• n_train_samples (int) – Number of training samples.

• n_val_samples (int) – Number of validation samples.

• height (int) – Sample image height in pixels. Default: 32.

• width (int) – Sample image width in pixels. Default: 32.

• n_classes (int) – Number of output classes.

• channel_means (list of float) – Channel means for normalization.

• channel_stds (list of float) – Channel stds for normalization.

• remote – S3 bucket or root directory where dataset is stored. Default: 's3://mosaicml-internal-dataset-cifar20'.

• name – Key used to determine where dataset is cached on local filesystem. Default: 'cifar20'.

• n_train_samples – Number of training samples. Default: 50000.

• n_val_samples – Number of validation samples. Default: 10000.

• n_classes – Number of output classes. Default: 20.

• channel_means – Channel means for normalization. Default: (0.5071, 0.4867, 0.4408).

• channel_stds – Channel stds for normalization. Default: (0.2675, 0.2565, 0.2761).

class composer.datasets.cifar.CIFARWebDatasetHparams(is_train=True, drop_last=True, shuffle=True, datadir=None, webdataset_cache_dir='/tmp/webdataset_cache/', webdataset_cache_verbose=False, shuffle_buffer=256, remote='', name='', n_train_samples=0, n_val_samples=0, height=32, width=32, n_classes=0, channel_means=(0, 0, 0), channel_stds=(0, 0, 0))

Bases: composer.datasets.hparams.WebDatasetHparams

Common functionality for CIFAR WebDatasets.

Parameters

• datadir (str) – The path to the data directory.

• is_train (bool) – Whether to load the training data or validation data. Default: True.

• drop_last (bool) – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle (bool) – Whether to shuffle the dataset. Default: True.

• datadir – The path to the data directory.

• is_train – Whether to load the training data or validation data. Default: True.

• drop_last – If the number of samples is not divisible by the batch size, whether to drop the last batch or pad the last batch with zeros. Default: True.

• shuffle – Whether to shuffle the dataset. Default: True.

• webdataset_cache_dir (str) – WebDataset cache directory.

• webdataset_cache_verbose (str) – WebDataset cache verbosity.

• remote (str) – S3 bucket or root directory where dataset is stored.

• name (str) – Key used to determine where dataset is cached on local filesystem.

• n_train_samples (int) – Number of training samples.

• n_val_samples (int) – Number of validation samples.

• height (int) – Sample image height in pixels. Default: 32.

• width (int) – Sample image width in pixels. Default: 32.

• n_classes (int) – Number of output classes.

• channel_means (list of float) – Channel means for normalization.

• channel_stds (list of float) – Channel stds for normalization.

initialize_object(batch_size, dataloader_hparams)

Creates a DataLoader or DataSpec for this dataset.

Parameters

• batch_size (int) – The size of the batch the dataloader should yield. This batch size is device-specific and already incorporates the world size.

• dataloader_hparams (DataLoaderHparams) – The dataset-independent hparams for the dataloader.

Returns

DataLoader or DataSpec – The DataLoader, or if the dataloader yields batches of custom types, a DataSpec.