-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
adjust Prodigy recipe to include new configs #68
Comments
Put in recipe for now and add comments noting that this is not long term. Also of note: turning off feed overlap in Ansible main config. |
The character-level highlighting and ignore button issues look good. But as far as I can tell from the test sessions, it's still serving out everything to everyone. Even after annotations are saved/submitted in two sessions, the other test sessions receive those same pages. |
@laurejt thanks for loading it with the smaller test set so we can try to test the two-annotators behavior. I'm still running into weird behavior. I opened up 3 different test sessions and each session was annotating different pages (there was no overlap), and I got a "no more tasks available" message after just 1 or 2 pages. |
For task routing to work properly when we want more than one annotation per task (i.e. page), we need to specify the session names up front. The distribution of annotations will only be approximately even (routing relies on mods of hashes). Work stealing has the potential to make this more uneven, since tasks can be "stolen" from one session and reassigned to another based on the relative age of the assignment. While this can make the work distribution more uneven, it will let us finish faster. For more on work stealing, see the answer I received from the Prodigy support forum. For the future annotations rounds and efforts, I'm investigating how we might guarantee 2 or more annotations per task without specifying session names. We can probably achieve this by writing a custom task router routine. That said, I've made a post asking Prodigy support on any recommendations for how to proceed. |
We'll have to decide what settings in the recipe versus environment.
For testing:
The text was updated successfully, but these errors were encountered: