- class streaming.StreamingDataLoader(*args, **kwargs)#
A streaming data loader.
Provides an additional checkpoint/resumption interface, for which it tracks the number of samples seen by the model this rank.
*args – List arguments.
**kwargs – Keyword arguments.
Load a dict containing training state (called from non-worker process).
This is called on each copy of the dataset when resuming.
obj (Dict[str, Any]) – The state.