InContextLearningLMAccuracy

class composer.metrics.InContextLearningLMAccuracy(dist_sync_on_step=False)

Computes accuracy for In-context learning (ICL) language modeling (LM) tasks.

ICL LM tasks consist of some number of example language modeling tasks (referred to as the ‘context’), followed by a test task where the model must correctly predict all the tokens following tokens in some passage (referred to as the ‘continuation’).

For example, the model may be provided the context below and evaluated on its ability to correctly predict the continuation. Note: it doesn’t matter whether the model correctly predicts the context tokens.

Context: The dog is->fuzzynthe water is->hotnthe tree is-> Continuation: green

Adds metric state variables:

correct (float): The number of instances where the prediction masked the target. total (float): The number of total instances that were predicted.

Parameters

dist_sync_on_step (bool, optional) – Synchronize metric state across processes at each forward() before returning the value at the step. Default: False.