class composer.metrics.InContextLearningMetric(*args, **kwargs)[source]#

Base class for In-context learning (ICL) metrics.

update(batch, output_logits=None, labels=None, outputs=None)[source]#

Abstract interface for computing an in-context learning metrics.

The output_logits argument is deprecated and will be removed in v0.22 while itโ€™s functionality will be moved to outputs.

  • batch (dict) โ€“ Batch must consist minimally of input_ids as well as any other structure needed to compute the metric.

  • output_logits (Tensor) โ€“ The model outputs evaluated on the batch input_ids

  • labels (Tensor) โ€“ The correct outputs.

  • outputs (Tensor) โ€“ The model outputs evaluated on the batch input_ids.


NotImplementedError โ€“ Abstract method must be implemented by subclasses