InContextLearningLMExpectedCalibrationError

class composer.metrics.InContextLearningLMExpectedCalibrationError(dist_sync_on_step=False, n_buckets=10)

Computes Expected Calibration Error (ECE) for In-context learning (ICL) language modeling (LM) tasks. (cite: https://arxiv.org/pdf/1706.04599.pdf).

For LM tasks, the model confidence is defined as the minimum probability assigned to all tokens in the continuation.

See InContextLearningExpectedCalibrationError for more info.