Return original text as tokens #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
johnmbw
force-pushed
the
return-original-text-as-tokens
branch
from
e27abcd to
dac0606
Compare
So that we can return that original tokens instead of the normlised ones.
johnmbw
force-pushed
the
return-original-text-as-tokens
branch
from
dac0606 to
180e699
Compare
johnmbw
force-pushed
the
return-original-text-as-tokens
branch
from
180e699 to
a65fdbc
Compare
ewencluley
reviewed
Oct 25, 2018
There was a problem hiding this comment.
While this looks fine, I'm struggling to understand fully why this is being done in the tokeniser. I think normally for the other tokenisers we normalise after tokenisation as a separate step. Is there a reason that cant be done here, leaving the tokeniser unchanged?
Ignore me, helps if i read the PR description 🤦♂️
| // kỹ would be normalised to kĩ internally | ||
| checkTokenization( | ||
| "Direct message để được chúng mình tư vấn kỹ hơn nhé", | ||
| "Direct","message", "để", "được", "chúng mình", "tư vấn", "kỹ", "hơn", "nhé" |
ewencluley
approved these changes
Oct 25, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The tokenizer was doing some normalisation of tokens (to match words to dictionary correctly I think), but those normalised tokens were being returned.
This change makes it return the original tokens (which could always be normalised again if wanted), so maintain the text that was in the source.
Also adds minimal tests etc