InContextLearningMetric

class composer.metrics.InContextLearningMetric(*args, **kwargs)

Base class for In-context learning (ICL) metrics.

update(batch, output_logits=None, labels=None, outputs=None)

Abstract interface for computing an in-context learning metrics.

The output_logits argument is deprecated and will be removed in v0.22 while it’s functionality will be moved to outputs.

Parameters

• batch (dict) – Batch must consist minimally of input_ids as well as any other structure needed to compute the metric.

• output_logits (Tensor) – The model outputs evaluated on the batch input_ids

• labels (Tensor) – The correct outputs.

• outputs (Tensor) – The model outputs evaluated on the batch input_ids.

Raises

NotImplementedError – Abstract method must be implemented by subclasses