DataloaderSpec

class composer.datasets.DataloaderSpec(dataset: torch.utils.data.dataset.Dataset[Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor]], drop_last: bool, shuffle: bool, collate_fn: Optional[Callable[[List[Any]], Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor]]] = None, worker_init_fn: Optional[Callable[[int], None]] = None, multiprocessing_context: Optional[Any] = None, generator: Optional[torch._C.Generator] = None, prefetch_fn: Optional[Callable[[Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor]], Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor]]] = None, split_fn: Callable[[Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor], int], List[Union[Tuple[Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]], Union[torch.Tensor, Tuple[torch.Tensor, ...], List[torch.Tensor]]], List[torch.Tensor], Dict[str, torch.Tensor], torch.Tensor]]] = <function _split_fn>)[source]

Bases: tuple

Specification for initializing a dataloader.

dataset

The initialized dataset from which to load data.

Type: Dataset

drop_last

Whether the final batch of an epoch should be discarded if there are fewer samples than the batch size.

Type: bool

shuffle

Whether the data should be shuffled.

Type: bool

collate_fn

A function to collate data before returning it from the dataloader.

Type: List[Any] -> Batch, optional

worker_init_fn

A function to be ran on each worker before dataloading begins.

Type: int -> None, optional

multiprocessing_context

The context to use for multiprocessing.

Type: Any, optional

generator

An RNG to be used for seeding workers.

Type: Generator, optional

prefetch_fn

A function to run for prefetching data.

Type: TPrefetchFn, optional

split_fn

A function to run to split batches into microbatches.

Type: Batch, int -> List[Batch]

collate_fn: Optional[Callable[[List[Any]], Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor]]]: Alias for field number 3

dataset: torch.utils.data.dataset.Dataset[Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor]]: Alias for field number 0

drop_last: bool: Alias for field number 1

generator: Optional[torch._C.Generator]: Alias for field number 6

multiprocessing_context: Any: Alias for field number 5

prefetch_fn: Optional[Callable[[Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor]], Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor]]]: Alias for field number 7

shuffle: bool: Alias for field number 2

split_fn: Callable[[Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor], int], List[Union[Tuple[Union[Tensor, Tuple[Tensor, ...], List[Tensor]], Union[Tensor, Tuple[Tensor, ...], List[Tensor]]], List[Tensor], Dict[str, Tensor], Tensor]]]: Alias for field number 8

worker_init_fn: Optional[Callable[[int], None]]: Alias for field number 4