class composer.metrics.InContextLearningLMAccuracy(dist_sync_on_step=False)[source]#

Computes accuracy for In-context learning (ICL) language modeling (LM) tasks.

ICL LM tasks consist of some number of example language modeling tasks (referred to as the โ€˜contextโ€™), followed by a test task where the model must correctly predict all the tokens following tokens in some passage (referred to as the โ€˜continuationโ€™).

For example, the model may be provided the context below and evaluated on its ability to correctly predict the continuation. Note: it doesnโ€™t matter whether the model correctly predicts the context tokens.

Context: The dog is->fuzzynthe water is->hotnthe tree is-> Continuation: green

Adds metric state variables:

correct (float): The number of examples where the model correctly predicted the whole continuation. total (float): The number of total examples seen.


dist_sync_on_step (bool, optional) โ€“ Synchronize metric state across processes at each forward() before returning the value at the step. Default: False.