build_streaming_cifar10_dataloader#

composer.datasets.build_streaming_cifar10_dataloader(global_batch_size, remote, *, local='/tmp/mds-cache/mds-cifar10', split='train', drop_last=True, shuffle=True, **dataloader_kwargs)[source]#

Builds a streaming CIFAR10 dataset

Parameters
  • global_batch_size (int) โ€“ Global batch size.

  • remote (str) โ€“ Remote directory (S3 or local filesystem) where dataset is stored.

  • local (str, optional) โ€“ Local filesystem directory where dataset is cached during operation. Defaults to '/tmp/mds-cache/mds-imagenet1k/`.

  • split (str) โ€“ Which split of the dataset to use. Either [โ€˜trainโ€™, โ€˜valโ€™]. Default: 'train`.

  • drop_last (bool, optional) โ€“ whether to drop last samples. Default: True.

  • shuffle (bool, optional) โ€“ whether to shuffle dataset. Defaults to True.

  • **dataloader_kwargs (Dict[str, Any]) โ€“ Additional settings for the dataloader (e.g. num_workers, etc.)