class composer.metrics.InContextLearningMultipleChoiceAccuracy(dist_sync_on_step=False)[source]#

Computes accuracy for In-context learning (ICL) multiple choice (MC) tasks.

ICL MC tasks consists of a series of questions with some number of possible choices (only one of which can be correct). At inference time each possible choice is given to the model as a separate input and the one for which the model assigns the lowest perplexity to the choice is considered the modelโ€™s choice. The model is correct if it โ€œchoosesโ€ the right answer.

Context: The dog is->fuzzynthe water is->hotnthe tree is-> Continuation: green

Adds metric state variables:

correct (float): The number of instances where the prediction masked the target. total (float): The number of total instances that were predicted.


dist_sync_on_step (bool, optional) โ€“ Synchronize metric state across processes at each forward() before returning the value at the step. Default: False.