a2ml tokenizer refactoring

Like the a2l tokenizer it now runs on bytes instead of chars: all relevant elements are ASCII; only comments could contain UTF-8. For readbility the code was split into several functions.
DanielT · Jun 21, 2024 · 311dc31 · 311dc31
1 parent 52fd8cb
commit 311dc31
Show file tree

Hide file tree

Showing 8 changed files with 441 additions and 305 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/a2lfile/Cargo.toml b/a2lfile/Cargo.toml
@@ -16,3 +16,5 @@ version = "2.0.0"
 [dependencies]
 thiserror = "1.0.50"
 
+[dev-dependencies]
+tempfile = "3.8"