composer.datasets.evaluator#

Specifies an instance of an Evaluator, which wraps a dataloader to include metrics that apply to a specific dataset.

Hparams

These classes are used with yahp for YAML-based configuration.

EvaluatorHparams

Params for the Evaluator.
class composer.datasets.evaluator.EvaluatorHparams(label, eval_dataset, metric_names=None)[source]#

Bases: yahp.hparams.Hparams

Params for the Evaluator.

Also see the documentation for the Evaluator.

Parameters

  • label (str) โ€“ Name of the Evaluator. Used for logging/reporting metrics.

  • eval_dataset (DatasetHparams) โ€“ Evaluation dataset.

  • metrics (list, optional) โ€“ List of strings of names of the metrics for the evaluator. Can be a torchmetrics.Metric name or the class name of a metric returned by metrics() If None, uses all metrics in the model. Default: None.

initialize_object(model, batch_size, dataloader_hparams)[source]#

Initialize an Evaluator

If the Evaluator metric_names is empty or None is provided, the function returns a copy of all the modelโ€™s default evaluation metrics.

Parameters

  • model (ComposerModel) โ€“ The model, which is used to retrieve metric names.

  • batch_size (int) โ€“ The device batch size to use for the evaluation dataset.

  • dataloader_hparams (DataLoaderHparams) โ€“ The hparams to use to construct a dataloader for the evaluation dataset.

Returns

Evaluator โ€“ The evaluator.