Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Pytorch MaxP Feature/ptmaxp #184

Merged
merged 51 commits into from
Aug 6, 2022
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
5f6ab05
first version of benchmark.eval with ir-measures
crystina-z Sep 2, 2021
018e5ca
benchmark.eval add relevance level support
crystina-z Sep 8, 2021
b0a4502
minor fix
crystina-z Sep 8, 2021
b25bbd4
remove msmarco-eval
crystina-z Sep 8, 2021
90920e9
clean
crystina-z Sep 8, 2021
2691d76
change all measures into str repr to avoid black problem
crystina-z Sep 8, 2021
ee3a0ef
skip evaluation if there is no matching qids
crystina-z Sep 9, 2021
f24bd3d
speed up training data prep - use set rather than list for train-qids…
crystina-z Sep 16, 2021
f915e45
add pt-maxp (train 30k + rerank top100: MRR@10=0.329)
crystina-z Sep 16, 2021
edace93
adapt config msmarco for pt monobert
crystina-z Sep 16, 2021
8cf1ef4
remove tqdm
crystina-z Sep 18, 2021
bd3d3aa
add decay into msmarco config
crystina-z Sep 18, 2021
2d41e28
fix import
crystina-z Sep 19, 2021
305ff92
add notes to ptmaxp
crystina-z Sep 25, 2021
16d6bdc
add shape for CE loss
crystina-z Sep 25, 2021
9c796c4
change sampling logic of pairsampler - sample one pos and neg at once…
crystina-z Sep 25, 2021
9d891d4
shuffle loaded tfrecord dataset
crystina-z Sep 25, 2021
18c144e
MSMARCO reproductino logs - nima
nimasadri11 Sep 26, 2021
ce4444d
Merge pull request #1 from nimasadri11/master
crystina-z Sep 26, 2021
ce392ac
tf amp: use both / None to align with pt
crystina-z Oct 2, 2021
23dcb3f
ms marco prepro doc; MRR@10=0.352 for pt-maxp; MRR@10=0.354 for tf-ma…
crystina-z Oct 2, 2021
addcb98
merge
crystina-z Oct 2, 2021
133de84
cross entropy; use avg rather than sum
crystina-z Oct 2, 2021
24cee86
support firstp, sump, avgp (same score on msp-v1)
crystina-z Oct 10, 2021
b5e7448
config for pt-maxp (rob04)
crystina-z Oct 10, 2021
9137bec
support eval dev and external runfile using external ckpt (dir)
crystina-z Oct 19, 2021
e788e9a
Update repro log for MS MARCO passage ranking task
leungjch Oct 20, 2021
1c570c3
Merge pull request #2 from leungjch/justin/update-repro-oct-19
crystina-z Oct 20, 2021
5d9fe65
Update msmarco reproduction log
edanerg Nov 5, 2021
c1bce9b
Fix markdown
edanerg Nov 5, 2021
65f0117
Merge branch 'feature/eval+ptmaxp' of github.com:crystina-z/capreolus…
crystina-z Nov 13, 2021
b730b98
add training flag to id2vec() to control different data format during…
crystina-z Nov 14, 2021
f2039ac
cleanup pt-maxp; mRR@10=0.352
crystina-z Nov 14, 2021
581ac27
Merge pull request #3 from AlexWang000/feature/eval+ptmaxp
crystina-z Jan 21, 2022
78d54be
revert the files that involving changing evaluation s.t. the PR isn't…
crystina-z May 8, 2022
7a7de77
merge with master
crystina-z May 8, 2022
3db0ff9
clean
crystina-z May 9, 2022
a87bfe7
adapt lce-passage extractor to the new extractor framework
crystina-z May 9, 2022
ef0f73d
make default msmarco-lce config a "small" version
crystina-z May 9, 2022
2edbb47
update repro doc
crystina-z May 9, 2022
38407df
update config msmarco
crystina-z May 10, 2022
10e0dc6
clean
crystina-z May 10, 2022
7a1ec64
first attmp to solve issue when warmup==epoch==1
crystina-z May 11, 2022
c263868
allow extrector to pad queries to the specified length
crystina-z May 11, 2022
db5e1ee
newline at the end of file
crystina-z May 11, 2022
ae536a5
black
crystina-z May 11, 2022
ea7e04a
dead code
crystina-z May 11, 2022
cdd90f3
bugfix
crystina-z May 11, 2022
95fd1d4
change the id2vec test case; so that the testing n-passage is 1
crystina-z May 11, 2022
30f3096
revert quick.md
crystina-z May 12, 2022
db0e405
for birch extractor; move the create_tf_train_feature and parse_tf_tr…
crystina-z May 12, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
change all measures into str repr to avoid black problem
crystina-z committed Sep 8, 2021
commit 2691d76722e03e70a6989b16b65dc5714f9bfad8
1 change: 0 additions & 1 deletion capreolus/evaluator.py
Original file line number Diff line number Diff line change
@@ -6,7 +6,6 @@

from capreolus.searcher import Searcher
from capreolus.utils.loginit import get_logger
from capreolus.eval.msmarco_eval import compute_metrics_from_files

from ir_measures import *
from ir_measures.measures import Measure
1 change: 1 addition & 0 deletions capreolus/task/rank.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from capreolus import ConfigOption, Dependency, evaluator
from capreolus.task import Task
from capreolus.utils.loginit import get_logger
from capreolus.evaluator import log_metrics_verbose
from capreolus.utils.trec import load_qrels, convert_metric, DEFAULT_METRICS

logger = get_logger(__name__) # pylint: disable=invalid-name
2 changes: 1 addition & 1 deletion capreolus/task/rerank.py
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@
from capreolus.searcher import Searcher
from capreolus.evaluator import log_metrics_verbose, format_metrics_string
from capreolus.task import Task
from capreolus.utils.trec import DEFAULT_METRICS
from capreolus.utils.trec import convert_metric, DEFAULT_METRICS
from capreolus.utils.loginit import get_logger

logger = get_logger(__name__)
7 changes: 4 additions & 3 deletions capreolus/trainer/pytorch.py
Original file line number Diff line number Diff line change
@@ -274,9 +274,10 @@ def train(self, reranker, train_dataset, train_output_path, dev_data, dev_output
# log dev metrics
metrics = benchmark.evaluate(preds, qrels)
logger.info("dev metrics: %s", format_metrics_string(metrics))
summary_writer.add_scalar("ndcg_cut_20", metrics[NDCG@20], niter)
summary_writer.add_scalar("map", metrics[AP], niter)
summary_writer.add_scalar("P_20", metrics[P@20], niter)
for metric_str in ["AP", "P@20", "NDCG@20"]:
metric = convert_metric(metric_str)
summary_writer.add_scalar(metric_str, metrics[metric], niter)

# write best dev weights to file
if metrics[metric] > dev_best_metric:
dev_best_metric = metrics[metric]
31 changes: 16 additions & 15 deletions capreolus/utils/trec.py
Original file line number Diff line number Diff line change
@@ -3,27 +3,28 @@
import xml.etree.ElementTree as ET
from collections import defaultdict
from ir_measures import *
from ir_measures.measures import Measure

from capreolus.utils.loginit import get_logger

logger = get_logger(__name__)

DEFAULT_METRICS = [
P@1,
P@5,
P@10,
P@20,
Judged@10,
Judged@20,
Judged@200,
AP,
NDCG@5,
NDCG@10,
NDCG@20,
Recall@100,
Recall@1000,
RR,
RR@10,
"P@1",
"P@5",
"P@10",
"P@20",
"Judged@10",
"Judged@20",
"Judged@200",
"AP",
"NDCG@5",
"NDCG@10",
"NDCG@20",
"R@100",
"R@1000",
"RR",
"RR@10",
]