EvalOutputLogging

class composer.callbacks.EvalOutputLogging(log_tokens=False, *args, **kwargs)

Logs eval outputs for each sample of each ICL evaluation dataset.

ICL metrics are required to support caching the model’s responses including information on whether model was correct. Metrics are responsible for returning the results of individual data points in a dictionary of lists. The callback will log the metric name, the depadded and detokenized input, any data stored in state.metric_outputs, and any keys from the batch passed into batch_keys_to_log. It will do so after every eval batch.