- class streaming.MDSWriter(*, columns, out, keep_local=False, compression=None, hashes=None, size_limit=67108864, **kwargs)#
Writes a streaming MDS dataset.
Output dataset directory to save shard files.
outis a local directory, shard files are saved locally.
outis a remote directory, a local temporary directory is created to cache the shard files and then the shard files are uploaded to a remote location. At the end, the temp directory is deleted once shards are uploaded.
outis a tuple of
(local_dir, remote_dir), shard files are saved in the local_dir and also uploaded to a remote location.
keep_local (bool) – If the dataset is uploaded, whether to keep the local dataset directory or remove it after uploading. Defaults to
compression (str, optional) – Optional compression or compression:level. Defaults to
hashes (List[str], optional) – Optional list of hash algorithms to apply to shard files. Defaults to
size_limit (Union[int, str], optional) – Optional shard size limit, after which point to start a new shard. If
None, puts everything in one shard. Can specify bytes human-readable format as well, for example
"100kb"for 100 kilobyte (100*1024) and so on. Defaults to
1 << 26.
**kwargs (Any) –
Additional settings for the Writer.
- progress_bar (bool): Display TQDM progress bars for uploading output dataset files to
a remote location. Default to
- max_workers (int): Maximum number of threads used to upload output dataset files in
parallel to a remote location. One thread is responsible for uploading one shard file to a remote location. Default to
min(32, (os.cpu_count() or 1) + 4).
Encode a joint shard out of the cached samples (single file).
bytes – File data.
Encode a sample dict to bytes.
sample (Dict[str, Any]) – Sample dict.
bytes – Sample encoded as bytes.
Get object describing shard-writing configuration.
Dict[str, Any] – JSON object.