Skip to content

Commit

Permalink
Added instructions on using the evaluation code.
Browse files Browse the repository at this point in the history
  • Loading branch information
arunchaganty-google committed May 17, 2023
1 parent fcf13a6 commit db633b0
Showing 1 changed file with 22 additions and 0 deletions.
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,25 @@ Here is an abbreviated example:
]
}
```

## Running the evaluation script

This repository also includes a standardized evaluation script that discounts
near-duplicate tracks in the evaluation metrics.

To use this script, format the model output in JSONL as follows (see
`model_outputs/` for examples):
```
{
"docid": "<dialog_id>:<turn_index>",
"neighbor": [
{ "docid": "<track_id>" }
...
]
}
```

You can then run the evaluation code as follows:
```
python3 eval.py --model_output <model_output.jsonl> --output output.csv
```

0 comments on commit db633b0

Please sign in to comment.