InContextLearningCodeEvalAccuracy

class composer.metrics.InContextLearningCodeEvalAccuracy(dist_sync_on_step=False)

Computes accuracy for In-context learning (ICL) code evaluation tasks.

ICL code eval tasks consist of some number of example code eval tasks (referred to as the ‘context’), followed by a test task where the model must complete the code, where we term the code completion a ‘continuation’.

In each case, the model constructs a given number of continuations (termed pass@K for K continuations), and each continuation is run against a set of test cases. The model is considered correct if at least one of the proposed continuations passes all the test cases.

Runs on AWS Lambdas by default.

Adds metric state variables:

correct (float): The number of instances where the predictions passed all the test cases. total (float): The number of total instances that were predicted.

Parameters

dist_sync_on_step (bool, optional) – Synchronize metric state across processes at each forward() before returning the value at the step. Default: False.

estimator(n, c, k)

Computes the pass@k metric.

Given the number of generated samples, n, the number of correct samples, c, and the k of interest, this function calculates pass@k as 1 - comb(n - c, k) / comb(n, k) as per the definition of pass@k in the HumanEval paper (https://arxiv.org/abs/2107.03374) and it’s associated implementation: https://github.com/openai/human-eval.

get_client()

Returns a client for the appropriate remote platform.

update(batch, outputs, labels)

Updates the pass@k accuracy of code generation.

Given a batch of prompts, test cases, and code generations, evaluates the code generations against the test cases and augments the pass@k accuracy of the batch to the values so far.

Parameters

• batch (Dict[str, Any]) – A batch of data produced by the InContextLearningCodeEvalDataset, with

• prompt (the) –

• cases (test) –

• following (and entry points. This will be a dictionary that must have the) –

• arguments –

• { – ‘prompts’: List[str], ‘test_inputs’: List[List[str]], ‘test_outputs’: List[List[str]], ‘entry_points’: List[str], ‘languages’: List[str], ‘generation_kwargs’: Dict[str, Any]

• } –

• outputs (List[str]) – A list of code generations in the format of HF generate with beam search,

• 2 (prompt 1 gen) –

• list (the) –

• 1 (prompt 2 gen) –

• 2 –

• 1 –

• 2] (prompt 2 gen) –

• labels (List[str]) – A list of the correct code generations, for compatibility with existing HF generate

• used. (functionalities. This is not) –