Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Parse HogQL with C++ for a huge speedup #17659

Merged
merged 78 commits into from
Oct 13, 2023
Merged
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
46d04cf
Add partial C++ parser of HogQL
Twixes Sep 28, 2023
a8dc5d4
Support all the rules and add proper error handling
Twixes Oct 2, 2023
e53ec36
Use `AlignAfterOpenBracket: BlockIndent`
Twixes Oct 2, 2023
3be9872
Reuse all the parser tests for the C++ backend
Twixes Oct 2, 2023
7906414
Fix relationship between headers and implementations
Twixes Oct 2, 2023
b866ebf
Add more error handling and fix minor issues
Twixes Oct 2, 2023
b35c93d
Build both Python and C++ parsers in the package.json script
Twixes Oct 2, 2023
f2a62ea
Update ARRAY JOIN error assertion
Twixes Oct 2, 2023
102d0a1
Improve timeit
Twixes Oct 2, 2023
6a695af
Move the C extension to the top level
Twixes Oct 2, 2023
ef5101a
Refactor `vector_to_list_string`
Twixes Oct 2, 2023
ec8ea8b
Build the parser on Linux
Twixes Oct 4, 2023
4af40cc
Build wheels for the parser
Twixes Oct 5, 2023
b5101ac
Simplify Linux build and fix macOS
Twixes Oct 5, 2023
ea137d5
Fix Homebrew paths on x86 and don't fail fast in CI
Twixes Oct 5, 2023
c261a81
Set MACOSX_DEPLOYMENT_TARGET for C++20
Twixes Oct 5, 2023
841bddb
Set up QEMU for Linux ARM builds
Twixes Oct 5, 2023
2a8d760
Publish the wheels on PyPI
Twixes Oct 5, 2023
c7dcefe
Avoiding Linux ARM emulation in CI for now
Twixes Oct 5, 2023
30a6257
Build sdist too
Twixes Oct 5, 2023
37ae412
Revert Dockerfile changes
Twixes Oct 5, 2023
6a4eece
Fix PyPI publish
Twixes Oct 5, 2023
5bf907b
Add README and optimize sdist build
Twixes Oct 5, 2023
9c4f952
Use setup.py directly instead of build
Twixes Oct 6, 2023
39f1b52
Use PyPI hogql-parser instead of local
Twixes Oct 6, 2023
d4cd499
Also revert production-unit.Dockerfile
Twixes Oct 6, 2023
feccbd2
Fix sdist upload and add Linux ARM back
Twixes Oct 6, 2023
c142308
No Linux ARM build in the end
Twixes Oct 6, 2023
db1b7d9
Fix artifact uploading
Twixes Oct 6, 2023
8c363d6
Do try building Linux ARM
Twixes Oct 6, 2023
e1453dd
Use `npm` in `grammar:build`
Twixes Oct 6, 2023
e194c89
Fix formatting of hogql_parser
Twixes Oct 6, 2023
e54387b
Build everything on macOS
Twixes Oct 6, 2023
c30afea
Revert "Build everything on macOS"
Twixes Oct 6, 2023
ca1fa3c
Use hogql-parser=0.1.1
Twixes Oct 6, 2023
7e10a55
Fix dylib in macOS wheel
Twixes Oct 6, 2023
713bfec
Bump hogql-parser version
Twixes Oct 6, 2023
bfed4f6
Fix missing module error
Twixes Oct 6, 2023
1214e21
Delete timeit.py
Twixes Oct 6, 2023
58df63b
Make error handling robust
Twixes Oct 6, 2023
62e7497
Format the C++
Twixes Oct 6, 2023
cc7f5c6
Use `hogql-parser==0.1.1`
Twixes Oct 9, 2023
66a980d
Fix reserved keyword error assertions
Twixes Oct 9, 2023
0e53256
Use HEAD hogql_paresr in CI
Twixes Oct 9, 2023
cff8a67
Fix `apt` usage
Twixes Oct 9, 2023
507b78b
Add some sudo in CI
Twixes Oct 9, 2023
bfe4603
Ensure package will be releasable before build
Twixes Oct 9, 2023
0d089c4
Bump version to 0.1.3
Twixes Oct 9, 2023
81d3c4b
Cover C++ `unquote_string` with tests
Twixes Oct 9, 2023
f3dfb89
Use BuildJet ARM runners for ARM builds
Twixes Oct 9, 2023
e8f94ed
Add some instructions
Twixes Oct 9, 2023
d0b6647
Add HogQL version check to backend CI
Twixes Oct 9, 2023
729a897
Update requirements.txt
Twixes Oct 9, 2023
826b9d4
Use `setuptools` instead of the deprecated `distutils`
Twixes Oct 9, 2023
1dfd70c
Fix working dir in backend CI
Twixes Oct 9, 2023
99c5bdf
Align ANTLR versions
Twixes Oct 10, 2023
2da9925
Add test for "mismatched input"
Twixes Oct 10, 2023
4922e64
Add types and bump version
Twixes Oct 10, 2023
3051b6e
Comment instead of failing version check
Twixes Oct 10, 2023
c50bf47
Automate hogql-release version bump
Twixes Oct 10, 2023
39edff4
Fix checkout token
Twixes Oct 10, 2023
954e1ac
Merge branch 'master' into turbo-parser
Twixes Oct 10, 2023
d3c8f34
Don't build hogql-parser if there were no changes
Twixes Oct 10, 2023
2b78d5d
Update query snapshots
github-actions[bot] Oct 10, 2023
2f28676
Update query snapshots
github-actions[bot] Oct 10, 2023
e0854eb
Update query snapshots
github-actions[bot] Oct 10, 2023
527ec7e
Update query snapshots
github-actions[bot] Oct 10, 2023
19617f5
Improve documentation
Twixes Oct 11, 2023
7a89bf2
Use new hogql-parser version
github-actions[bot] Oct 11, 2023
4b5e8a4
Fix error start and end initialization
Twixes Oct 12, 2023
4096807
Note `antlr4-cpp-runtime`
Twixes Oct 12, 2023
5e5ba40
Also remove NUL chars in C++
Twixes Oct 12, 2023
debb5ed
Check ANTLR4 runtime archive checksum for security
Twixes Oct 12, 2023
640c159
Note more decrefs to add
Twixes Oct 12, 2023
eca0d64
Add vector size checks
Twixes Oct 12, 2023
6015bc4
Use new hogql-parser version
github-actions[bot] Oct 12, 2023
4d4d6c0
Don't support the `start` arg in C++ `parse_expr`
Twixes Oct 13, 2023
29980df
Use new hogql-parser version
github-actions[bot] Oct 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert Dockerfile changes
Twixes committed Oct 10, 2023
commit 37ae41276c3ab34f96883e8a988c7ede964d43cf
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -23,7 +23,6 @@
!frontend/public
!frontend/*.*
!ee
!hogql_parser
!.devcontainer
!plugin-server/package.json
!plugin-server/pnpm-lock.yaml
30 changes: 11 additions & 19 deletions production.Dockerfile
Original file line number Diff line number Diff line change
@@ -88,27 +88,19 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Compile and install Python dependencies.
# We install those dependencies on a custom folder that we will
# then copy to the last image.
COPY requirements.txt ./
RUN apt-get update && \
apt-get install -y --no-install-recommends \
"build-essential" "git" "libpq-dev" "libxmlsec1" "libxmlsec1-dev" "libboost-all-dev" "libffi-dev" "curl" "unzip" \
"cmake" "uuid-dev" "pkg-config" && \
rm -rf /var/lib/apt/lists/*
RUN curl https://www.antlr.org/download/antlr4-cpp-runtime-4.13.1-source.zip --output antlr4-source.zip && \
unzip antlr4-source.zip -d antlr4-source && \
cd antlr4-source && \
mkdir build && \
mkdir run && \
cd build && \
cmake .. && \
DESTDIR=../run make install && \
cd ../run/usr/local/include && \
cp -r antlr4-runtime /usr/local/include && \
cd ../lib && \
cp libantlr4-runtime.so libantlr4-runtime.so.* /usr/local/lib && \
ldconfig
COPY requirements.txt ./
COPY hogql_parser hogql_parser/
RUN pip install -r requirements.txt --compile --no-cache-dir --target=/python-runtime
"build-essential" \
"git" \
"libpq-dev" \
"libxmlsec1" \
"libxmlsec1-dev" \
"libffi-dev" \
"pkg-config" \
&& \
rm -rf /var/lib/apt/lists/* && \
pip install -r requirements.txt --compile --no-cache-dir --target=/python-runtime

ENV PATH=/python-runtime/bin:$PATH \
PYTHONPATH=/python-runtime