How does tokenize separate SourceLines?

[Eisverygoodletter]

The function tokenize outputs an array of SourceLines. Is it guaranteed that if i provide it with Text with some multiline code, every line's tokens will be mapped to one SourceLine without any formatting/line changing behaviour?

For example, if I had this single line of oddly written piece of C code:

int do_something() { call_a_function(); int some_number = 3; return 123; }

Would very token (e.g. int, do_something, (), {, call_a_function ...) will end up on the same SourceLine with every token in sequence?

[jgm]

Yes -- why don't you simply try it?