Handle newlines during string parsing while lexing

[nirmal-j-patel]

If newline strings are encountered while lexing, the lexer now handles newline characters by incrementing current line number. This provides correct line number when displaying errors. If the lexer encounters end of file before string end, then it will use the start of the string as the location to an report error.

Fixes #2187

[nirmal-j-patel]

https://github.com/Rust-GCC/gccrs/pull/2684/files#diff-bf317668e350e4e1bb830fab7e19d5e0b96e95471067d2d6b1bd6dabb95ff0aeR2061

I am not sure if we still this. When we encounter a string spanning multiple lines, do we still want to add length of string to current_column?

[P-E-P]

https://github.com/Rust-GCC/gccrs/pull/2684/files#diff-bf317668e350e4e1bb830fab7e19d5e0b96e95471067d2d6b1bd6dabb95ff0aeR2061

I am not sure if we still this. When we encounter a string spanning multiple lines, do we still want to add length of string to current_column?

I'm not well versed in this part of the code but it seems like current_column belongs to the public internal state of the lexer, someone might depend on it after calling this function am I right ?

[nirmal-j-patel]

It appears that if we continue to add string length to its beginning locus, then it will give wrong location when showing errors involving multi-line strings. This is because the length of the multiple line string will be longer than the length of any particular line. I am incrementing current_column per character and resetting it to 1 on linebreak. I am not sure how it would work with unicode characters. However, removing this part works better from what I can tell.

Without removing current_column += length;

/home/user/projects/gccrs_test/location_error_str.rs:3:20: error: unicode escape should start with ‘{’
    3 | " + "\uu";
      |                    ^
/home/user/projects/gccrs_test/location_error_str.rs:6:4: error: unrecognised token ‘identifier’ for start of item
    6 |    ERROR_TIME
      |    ^~~~~~~~~~
/home/user/projects/gccrs_test/location_error_str.rs:6:4: error: failed to parse item in crate

After removing current_column += length;

/home/user/projects/gccrs_test/location_error_str.rs:3:5: error: unicode escape should start with ‘{’
    3 | " + "\uu";
      |     ^
/home/user/projects/gccrs_test/location_error_str.rs:6:4: error: unrecognised token ‘identifier’ for start of item
    6 |    ERROR_TIME
      |    ^~~~~~~~~~
/home/user/projects/gccrs_test/location_error_str.rs:6:4: error: failed to parse item in crate

[CohenArthur]

I think this is good. I'd like to see maybe one more test to make sure we parse the newlines properly, so maybe something that calls printf and checks the output with dg-output?

[nirmal-j-patel]

Apologies for the delay. How does it look now?

[P-E-P]

It looks good 🙂