InContextLearningMetric

class composer.metrics.InContextLearningMetric(**kwargs)

update(batch, output_logits, labels)

Abstract interface for computing an in-context learning metrics.

Parameters

• batch (dict) – Batch must consist minimally of input_ids as well as any other structure needed to compute the metric.

• output_logits (Tensor) – The model outputs evaluated on the batch input_ids

• labels (Tensor) – The correct outputs.

Raises

NotImplementedError – Abstract method must be implemented by subclasses