Consider tweaking tokenization

[hoelzro]

I might want to tweak how the plugin uses lunr to tokenize things, to handle hyphenated words or URLs.

Examples:

#5 (comment)

tw-full-text-search/tests/test-simple.js

Lines 269 to 281 in 9d383ac

	xit('should pick up "twitter" in a URL', async function() {
	await prepare();
	var text = 'https://twitter.com/hoelzro/status/877901644125663232';
	$tw.wiki.addTiddler(new $tw.Tiddler(
	$tw.wiki.getCreationFields(),
	{ title: 'ContainsTweetLink', type: 'text/vnd.tiddlywiki', text: text },
	$tw.wiki.getModificationFields()
	));
	await waitForNextTick();
	var results = wiki.compileFilter('[ftsearch[twitter]]')();
	expect(results).toContain('ContainsTweetLink');
	});

[hoelzro]

Another interesting data point for this: e-mail is treated as two tokens, which kind of screws things up

Would it make sense just to use a tokenizer that recognizes certain exceptions (like e-mail) and certain special prefixes (like re-)? Alternative to a list of exceptions, we could have logic that bundles prefixes of a certain length (eg. 3 or fewer characters)