Experiment Reports and Level 1 Assertions @ Runtime

[jlewi]

Experiment Report

After running an evaluation experiment, we compute a report that contains the key metrics we want to track. To start with this is

• Number of cell match results
• Number of errors and examples
• Generate latency measured as percentiles
• Level1 assertion stats

Level 1 Assertion stats

• Add a level 1 assertion to test whether the document is zero.
  • I believe I observed this started happening when we included a fix to outputs not being included (Cell Outputs aren't actually included in the LLM request. #286) in Start generating markup cells #285.
  • I think the problem is the cell outputs could be very long and this could end up eating all the available context buffer

Reintegrate Level 1 Assertions Into Evaluation

• Fix Integrate Level 1 Assertions Into Evaluator #261
• We start computing level 1 assertions at RunTime so that they are available in production and evaluation
• Level1 assertions are computed and then logged
• Our Analyzer pipeline reads the assertions from the logs and adds them to the trace
• Our evaluation report accumulates assertion statistics and reports them

[netlify]

✅ Deploy Preview for foyle canceled.

Name	Link
🔨 Latest commit	a2a6cd1
🔍 Latest deploy log	https://app.netlify.com/sites/foyle/deploys/670e9c4669bedc0008f02bdd