InContextLearningMCExpectedCalibrationError#

class composer.metrics.InContextLearningMCExpectedCalibrationError(dist_sync_on_step=False, n_buckets=10)[source]#

Computes Expected Calibration Error (ECE) for In-context learning (ICL) multiple choice (MC) tasks. (source: https://arxiv.org/abs/2012.00955).

For MC tasks, the model confidence is defined as the softmax of average per-token probability assigned to the top question choice.

See InContextLearningExpectedCalibrationError for more info.