composer.metrics#

A collection of common torchmetrics.

Classes

BinaryF1Score

Implements F1 Scores for binary classification tasks via sklearn.

CrossEntropy

Torchmetrics cross entropy loss implementation.

Dice

The Dice Coefficient for evaluating image segmentation.

InContextLearningCodeEvalAccuracy

Computes accuracy for In-context learning (ICL) code evaluation tasks.

InContextLearningLMAccuracy

Computes accuracy for In-context learning (ICL) language modeling (LM) tasks.

InContextLearningLMExpectedCalibrationError

Computes Expected Calibration Error (ECE) for In-context learning (ICL) language modeling (LM) tasks.

InContextLearningMCExpectedCalibrationError

Computes Expected Calibration Error (ECE) for In-context learning (ICL) multiple choice (MC) tasks.

InContextLearningMetric

Base class for In-context learning (ICL) metrics.

InContextLearningMultipleChoiceAccuracy

Computes accuracy for In-context learning (ICL) multiple choice (MC) tasks.

InContextLearningQAAccuracy

Computes accuracy for In-context learning (ICL) question answering (QA) tasks.

LanguageCrossEntropy

Torchmetric that computes cross entropy on language modeling outputs.

LanguagePerplexity

Subclasses LanguageCrossEntropy to implement perplexity.

LossMetric

Turns a torch.nn Loss Module into distributed torchmetrics Metric.

MAP

Computes the Mean-Average-Precision (mAP) and Mean-Average-Recall (mAR) for object detection predictions.

MIoU

Torchmetrics mean Intersection-over-Union (mIoU) implementation.

MaskedAccuracy

Computes accuracy with support for masked indices.