Any documentation on how tokenization interacts with rules?

[bobbiec]

I recently had to debug an issue where I wanted to disallow the word I, but allow I/O (among other things). I ended up with the following incorrect rule:

 extends: existence
 message: "Don't use first person ('%s')"
 nonword: true
 level: warning
 ignorecase: true
 tokens:
 - \bI\b(?!\/o)
 - (?<!contact )us\b
 # etc

Which ended up producing a bunch of confusing false positives, including things like the i in <td class="icon"><i class="fa icon-important" title="Important"></i></td>, which was baffling because I thought that scope: raw would be required to match that. Eventually, I fixed the issue by guesswork, but it took a long time to figure out why, because I wasn't sure even after poking the source code a bit:

1. What is the final regex produced?
2. What unit of text does the regex match against?

I figured out that I could get the answer to (1) by putting a dangling backslash at the end of tokens, which causes the regex compile to fail and output the string. But it would be useful to be able to view debug logging for the rules so I don't have to do that...

I'm still not sure of the answer to (2). Any advice?

[jdkato]

Neither of these questions have a general answer.

What is the final regex produced?

It depends on the content and extends value of your rule. The process for existence is described here. You can also use Vale Studio to view the final regex.

What unit of text does the regex match against?

This depends on the value of scope. By default, it is all text with markup syntax removed.