Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docs] Add NanoBEIR to the Training Overview evaluators #3089

Merged
merged 2 commits into from
Nov 26, 2024
Merged

[docs] Add NanoBEIR to the Training Overview evaluators #3089

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c7e432d
Add NanoBEIR to the Training Overview evaluators
tomaarsen Nov 26, 2024
fa6bf0b
Use dots for all Required Data sentences
tomaarsen Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions docs/sentence_transformer/training_overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,10 @@ Here are the implemented Evaluators that come with Sentence Transformers:
======================================================================== ===========================================================================================================================
Evaluator Required Data
======================================================================== ===========================================================================================================================
:class:`~sentence_transformers.evaluation.BinaryClassificationEvaluator` Pairs with class labels
:class:`~sentence_transformers.evaluation.EmbeddingSimilarityEvaluator` Pairs with similarity scores
:class:`~sentence_transformers.evaluation.InformationRetrievalEvaluator` Queries (qid => question), Corpus (cid => document), and relevant documents (qid => set[cid])
:class:`~sentence_transformers.evaluation.BinaryClassificationEvaluator` Pairs with class labels.
:class:`~sentence_transformers.evaluation.EmbeddingSimilarityEvaluator` Pairs with similarity scores.
:class:`~sentence_transformers.evaluation.InformationRetrievalEvaluator` Queries (qid => question), Corpus (cid => document), and relevant documents (qid => set[cid]).
:class:`~sentence_transformers.evaluation.NanoBEIREvaluator` No data required.
:class:`~sentence_transformers.evaluation.MSEEvaluator` Source sentences to embed with a teacher model and target sentences to embed with the student model. Can be the same texts.
:class:`~sentence_transformers.evaluation.ParaphraseMiningEvaluator` Mapping of IDs to sentences & pairs with IDs of duplicate sentences.
:class:`~sentence_transformers.evaluation.RerankingEvaluator` List of ``{'query': '...', 'positive': [...], 'negative': [...]}`` dictionaries.
Expand Down Expand Up @@ -363,6 +364,27 @@ Sometimes you don't have the required evaluation data to prepare one of these ev
# You can run evaluation like so:
# dev_evaluator(model)

.. tab:: NanoBEIREvaluator

.. raw:: html

<div class="sidebar">
<p class="sidebar-title">Documentation</p>
<ul class="simple">
<li><a class="reference internal" href="../package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.NanoBEIREvaluator" title="sentence_transformers.evaluation.NanoBEIREvaluator"><code class="xref py py-class docutils literal notranslate"><span class="pre">sentence_transformers.evaluation.NanoBEIREvaluator</span></code></a></li>
</ul>
</div>

::

from sentence_transformers.evaluation import NanoBEIREvaluator

# Initialize the evaluator. Unlike most other evaluators, this one loads the relevant datasets
# directly from Hugging Face, so there's no mandatory arguments
dev_evaluator = NanoBEIREvaluator()
# You can run evaluation like so:
# dev_evaluator(model)

.. warning::

When using `Distributed Training <training/distributed.html>`_, the evaluator only runs on the first device, unlike the training and evaluation datasets, which are shared across all devices.
Expand Down
Loading