chore(hogql): Full error & NULL handling in C++ parser (1.0.0) #18240
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This is the final stage of making the C++ parser production-ready: to prevent any and all segfaults, we must check for
NULL
pointers in all Python/C API call returningPyObject*
, and for -1 in all calls returning an integer (e.g.PyList_Size()
, but alsoPyObject_SetAttrString()
).I had to do a bunch more reading of CPython docs and code to get all of this right, which is why this is not the way I initially wrote the converter… even though that would have made for much more readable PRs than this one.
The way I went about making sure nothing is missed is basically just checking every match for this regex:
(Py(Object_|Unicode_|Dict_|List_|Long_|Float_|Tuple_|Import_|_(Va)?Build))|build_ast_node\(|is_ast_node_instance\(|get_ast_enum_member\(
I had to rewrite nearly each of the 200+ call sites to handle every situation where a NULL might pop up. This means throwing an error after decrementing refcounts for all
PyObject*
s in scope.Additionally, I made sure we similarly clean up refcounts if any
visit()
call throws (which can be due to a Python exception, but also a syntax exception).There's some new info on all of the above in
hogql_parser/CONTRIBUTING.md
.How did you test this code?
All parser tests succeed.