- class composer.callbacks.MLPerfCallback(root_folder, index, benchmark='resnet', target=0.759, division='open', metric_name='MulticlassAccuracy', metric_label='eval', submitter='MosaicML', system_name=None, status='onprem', cache_clear_cmd=None, host_processors_per_node=None, exit_at_target=False)#
Create compliant results file for MLPerf Training benchmark.
A submission folder structure will be created with the
root_folderas the base and the following directories:
root_folder/ results/ [system_name]/ [benchmark]/ results_0.txt results_1.txt ... systems/ [system_name].json
A required systems description will be automatically generated, and best effort made to populate the fields, but should be manually checked prior to submission.
Currently, only open division submissions are supported with this Callback.
from composer.callbacks import MLPerfCallback callback = MLPerfCallback( root_folder='/submission', index=0, metric_name='MulticlassAccuracy', metric_label='eval', target='0.759', )
During training, the metric found in
state.eval_metrics[evaluator_label][metric_name]will be compared against the target criterion.
This is currently an experimental logger that has not been used (yet) to submit an actual result to MLPerf. Please use with caution.
MLPerf submissions require clearing the system cache prior to any training run. By default, this callback does not clear the cache, as that is a system specific operation. To enable cache clearing, and thus pass the mlperf compliance checker, provide a
cache_clear_cmdthat will be executed with
root_folder (str) – The root submission folder
index (int) – The repetition index of this run. The filename created will be
benchmark (str, optional) – Benchmark name. Currently only
target (float, optional) – The target metric before the mllogger marks the stop of the timing run. Default:
division (str, optional) – Division of submission. Currently only
opendivision supported. Default:
metric_name (str, optional) – name of the metric to compare against the target. Default:
metric_label (str, optional) – The label name. The metric will be accessed via
submitter (str, optional) – Submitting organization. Default:
system_name (str, optional) – Name of the system (e.g. 8xA100_composer). If not provided, system name will default to
status (str, optional) – Submission status. One of (onprem, cloud, or preview). Default:
cache_clear_cmd (str, optional) – Command to invoke during the cache clear. This callback will call
os.system(cache_clear_cmd). Default is disabled (None)
host_processors_per_node (int, optional) – Total number of host processors per node. Default:
exit_at_target (bool, optional) – Whether to exit training when target metric is met. Default: