class composer.callbacks.EvalOutputLogging(log_tokens=False, *args, **kwargs)[source]#

Logs eval outputs for each sample of each ICL evaluation dataset.

ICL metrics are required to support caching the modelโ€™s responses including information on whether model was correct. Metrics are responsible for returning the results of individual data points in a dictionary of lists. The callback will log the metric name, the depadded and detokenized input, any data stored in state.metric_outputs, and any keys from the batch passed into batch_keys_to_log. It will do so after every eval batch.