avoid allocating hats to the first letter of a token

[josharian]

We could get much fancier than this,
but after running this with a day it appears to help some,
and it is nice and simple.

I propose that we declare that it fixes #1658,
at least for now.

Checklist

• [/] I have added tests
• [/] I have updated the docs and cheatsheet
• [/] I have not broken the cheatsheet

[josharian]

I plan to keep running this for a little while longer, gathering data, but I thought I would share it in case anyone else wants to play with it.

(I know the tests are busted.)

[pokey]

Looks good with minor tweak

[pokey]

This doesn't feel like the right place for this logic; seems more specific to hat allocator. I'd be inclined to inject it right after we call getTokenGraphemes in getTokenRemainingHatCandidates and add it to HatCandidate.

I'd also be tempted to handle camel case / snake case, but yeah prob not necessary in this draft? Shouldn't be too hard, tho; just for each char

1. if it's lowercase letter:
   a. return false if prev char exists and is letter
   b. else return true
2. else if it's uppercase letter:
   a. return true if previous char doesn't exist or is not uppercase letter
   b. else return false
3. else return false

I'd also advise using Unicode character classes, eg \p{Lu}, \p{L}, \P{L} etc

Also, it looks like your code should handle the case of the same grapheme appearing twice in the same token well, right? Ie we'll end up with two candidates, and end up preferring the one that is not first in its token, right?

[josharian]

Word splitting (for brains) is subtle. One case this would get wrong is languages that prefer all-caps initialisms, like Go and Swift. Consider URLRequest: The second R begins a word, but would be marked as not-initial by your heuristic.

I was also worried about whether we needed to respect the settings from the user about how to tokenize.

But the perfect is the enemy of the good! I'll implement your suggestion and play with it.

[josharian]

I'd be inclined to inject it right after we call getTokenGraphemes in getTokenRemainingHatCandidates and add it to HatCandidate.

That seems reasonable. Will play with that. It'll spare me fixing 94 tests. :)

[josharian]

Another hard word-splitting case: thisIsATest

[josharian]

I'd be inclined to inject it right after we call getTokenGraphemes in getTokenRemainingHatCandidates

I forgot. At that point we've normalized the grapheme, so we no longer have case information. We could add an original text field to the graphemes. Or punt.

(The normalization is also the reason that I used [a-z] instead of unicode character classes. But I'll happily switch back on that front.)

[pokey]

Ahh right. Keep in mind tho that the normalisation won't always strip accents; the user can customise normalisation. You can use the offsets to get the original text

Your counterexamples for splitting are interesting, maybe better to just keep your heuristic?

But yeah I wonder if it's worth all the trouble 🤔

Maybe let's discuss at meet-up

[josharian]

here's another rev. lots of tests are still failing; it's going to be tedious to fix them, so I'd like to wait until we are relatively confident in the rest of the direction.

[josharian]

notes to self:

• correctly handle _abcTest (are we avoiding _ or a?)
• perf test
• maybe re-use tokenizers
• switch to ranges
• tests: stats, fixtures
• data gathering for end users
  • no phones/replace
  • jsonl
  • open append/exclusive
  • command payload
  • rotate monthly
  • include extension version

[pokey]

update: @AndreasArvidsson is going to have a look and take this one home if it's pretty close to mergeable in its current form

[josharian]

update: @AndreasArvidsson is going to have a look and take this one home if it's pretty close to mergeable in its current form

great, thanks!

[AndreasArvidsson]

@josharian Have you evaluated the difference between just avoiding the first character in the token verses the first character in every subword? When I first thought about this problem I kinda just envisioned the first character in the token, but your implementation is doing every subword which could be better. Any insight?

[josharian]

I remember thinking at the time that doing sub words was important. But It is not something I ever gathered data about, because the effects are purely qualitative. And a lot of time has now gone by…