For specific lib versions, see requirements.txt
and install them with
pip install -r requirements.txt
The code runs on:
CPU Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
GPU NVIDIA Tesla V100-PCIE-16GB
RAM 16GB
HDD 16GB
Each directory bert
, bilstm
and linear_models
contains a baseline model.
Generally, a model sould produce output directory (e.g. bert/out
) with result jsonl
files named after the task name (e.g. RuMedTop3.jsonl
).
Each file contains same samples as in test parts enhanced with the prediction
field.
Examples,
for RuMedTop3.jsonl
{
"idx": "qaf1454f",
"code": "I11",
"prediction": ["I11", "I20", "I10"]
}
or RuMedSymptomRec.jsonl
{
"idx": "q45f6321",
"code": "боль в шее",
"prediction": ["тошнота", "боль в шее", "частые головные боли"]
}
or RuMedDaNet.jsonl
{
"pairID": "f5309eadb4eacf0f144b24e260643ea2",
"answer": "да",
"prediction": "нет"
}
or RuMedNLI.jsonl
{
"pairID": "1f2a8146-66c7-11e7-b4f2-f45c89b91419",
"gold_label": "entailment",
"prediction": "neutral"
}
or RuMedNER.jsonl
{
"idx": "769708.tsv_5",
"ner_tags": ["B-Drugname", "O", "B-Drugclass", "O", "O"],
"prediction": ["B-Drugclass", "O", "O", "O", "O"]
}
It is the script used to prepare data for the benchmark tasks from raw data files.
python tasks_builder.py
It is the script to evaluate the test results.
Run it like
python eval.py --out_dir bert/out
or
python eval.py --out_dir human