Fix KeyError encountered with some texts
#181
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neiljdo
force-pushed
the
break-cycles-fix
branch
from
b9c9b2e to
f250645
Compare
neiljdo
changed the title
KeyError encountered with some texts
|
@dimkart I've already marked this as ready for review but we're still waiting for the updated simplebooks coverage test. @nikhilkhatri I've implemented an update to the reader, hence the review request. PTAL! 🙏🏼 |
| reindexer = {} | ||
| j = 0 | ||
| for i, otok in enumerate(parsed_toks): |
There was a problem hiding this comment.
@neiljdo in what situation are an enumeration and parsed_tok_indxs going to disagree? I thought in the tree they're guaranteed to be [0 ... n-1]?
There was a problem hiding this comment.
@nikhilkhatri it happens when the tree is modified after, e.g. when adding the '' word for missing nouns. The '' was never in the original tokens. These were what I saw when investigating the sentence Anna gave. You can use the following example:
text112 = "Gently and Bacchus delve into a world of army secrets when a young former soldier, Scott Tanner, commits a murder in a Turkish bath. Whilst investigating Tanner's history, Gently hears of horrific allegations of what some soldiers have to face from their own side. He is forced to question the uncomfortable truth of what it means to serve one's queen and country, as an event from the past presses on his conscience."
dimkart
approved these changes
Nov 14, 2024
|
@neiljdo Let's merge after the end of coverage tests. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #180.
There are two causes to the
KeyError:break_cycles=True''word placeholder when reindexing. This is fixed by pairing the tokens with their token index, obtained viatree.get_word_indices(), instead of just enumeratingtree.get_words()