Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rust funnel udfs (off of master) #25263

Closed
wants to merge 10 commits into from
Closed

feat: rust funnel udfs (off of master) #25263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ff7d4e7
funnel udfs python
aspicer Oct 1, 2024
5ee52e8
rust funnel udfs
aspicer Oct 1, 2024
e3cfea3
remove alpine
aspicer Oct 1, 2024
97d1af8
glibc instead of musl
aspicer Oct 2, 2024
86e70e1
libc6
aspicer Oct 2, 2024
d760efd
python 3.8
aspicer Oct 2, 2024
fcebd85
python 3.9
aspicer Oct 2, 2024
32f83a0
rerun versioner
aspicer Oct 2, 2024
403ac62
Update query snapshots
github-actions[bot] Oct 2, 2024
6d7dec4
Update query snapshots
github-actions[bot] Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/run-backend-tests/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ runs:

- name: Upload updated timing data as artifacts
uses: actions/upload-artifact@v4
if: ${{ inputs.person-on-events != 'true' && inputs.clickhouse-server-image == 'clickhouse/clickhouse-server:23.12.6.19-alpine' }}
if: ${{ inputs.person-on-events != 'true' && inputs.clickhouse-server-image == 'clickhouse/clickhouse-server:23.12.6.19' }}
with:
name: timing_data-${{ inputs.segment }}-${{ inputs.group }}
path: .test_durations
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci-backend-update-test-timing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@ jobs:
group: 1
token: ${{ secrets.POSTHOG_BOT_GITHUB_TOKEN }}
python-version: '3.11.9'
clickhouse-server-image: 'clickhouse/clickhouse-server:23.12.6.19-alpine'
clickhouse-server-image: 'clickhouse/clickhouse-server:23.12.6.19'
segment: 'FOSS'
person-on-events: false

- name: Upload updated timing data as artifacts
uses: actions/upload-artifact@v4
if: ${{ inputs.person-on-events != 'true' && inputs.clickhouse-server-image == 'clickhouse/clickhouse-server:23.12.6.19-alpine' }}
if: ${{ inputs.person-on-events != 'true' && inputs.clickhouse-server-image == 'clickhouse/clickhouse-server:23.12.6.19' }}
with:
name: timing_data-${{ inputs.segment }}-${{ inputs.group }}
path: .test_durations
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci-backend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ jobs:
fail-fast: false
matrix:
python-version: ['3.11.9']
clickhouse-server-image: ['clickhouse/clickhouse-server:23.12.6.19-alpine']
clickhouse-server-image: ['clickhouse/clickhouse-server:23.12.6.19']
segment: ['Core']
person-on-events: [false, true]
# :NOTE: Keep concurrency and groups in sync
Expand All @@ -249,7 +249,7 @@ jobs:
include:
- segment: 'Temporal'
person-on-events: false
clickhouse-server-image: 'clickhouse/clickhouse-server:23.12.6.19-alpine'
clickhouse-server-image: 'clickhouse/clickhouse-server:23.12.6.19'
python-version: '3.11.9'
concurrency: 1
group: 1
Expand Down Expand Up @@ -320,7 +320,7 @@ jobs:
strategy:
fail-fast: false
matrix:
clickhouse-server-image: ['clickhouse/clickhouse-server:23.12.6.19-alpine']
clickhouse-server-image: ['clickhouse/clickhouse-server:23.12.6.19']
if: needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.base.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ services:
# Note: please keep the default version in sync across
# `posthog` and the `charts-clickhouse` repos
#
image: ${CLICKHOUSE_SERVER_IMAGE:-clickhouse/clickhouse-server:23.12.6.19-alpine}
image: ${CLICKHOUSE_SERVER_IMAGE:-clickhouse/clickhouse-server:23.12.6.19}
restart: on-failure

zookeeper:
Expand Down
8 changes: 7 additions & 1 deletion docker/clickhouse/docker-entrypoint-initdb.d/init-db.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
#!/bin/bash
set -e

apk add python3
apt-get update
# Necessary because clickhouse runs on Ubuntu 20, which has an old glibc and an old python default
# Can remove when we upgrade clickhouse, as the new images run on Ubuntu 22
apt-get -y install python3.9
ln -s /usr/bin/python3.9 /usr/bin/python3
wget http://launchpadlibrarian.net/588931980/libc6_2.35-0ubuntu3_amd64.deb
dpkg --auto-deconfigure -i libc6_2.35-0ubuntu3_amd64.deb
cp -r /idl/* /var/lib/clickhouse/format_schemas/
28 changes: 18 additions & 10 deletions docker/clickhouse/user_defined_function.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<function>
<type>executable</type>
<name>aggregate_funnel</name>
<return_type>Array(Tuple(Int8, Nullable(String), Array(Float64)))</return_type>
<return_type>Array(Tuple(Int8, Nullable(String), Array(Float64), Array(Array(UUID))))</return_type>
<return_name>result</return_name>
<argument>
<type>UInt8</type>
Expand All @@ -25,17 +25,18 @@
<name>prop_vals</name>
</argument>
<argument>
<type>Array(Tuple(Nullable(Float64), Nullable(String), Array(Int8)))</type>
<type>Array(Tuple(Nullable(Float64), UUID, Nullable(String), Array(Int8)))</type>
<name>value</name>
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel.py</command>
<command>aggregate_funnel</command>
<lifetime>600</lifetime>
</function>

<function>
<type>executable</type>
<name>aggregate_funnel_cohort</name>
<return_type>Array(Tuple(Int8, UInt64, Array(Float64)))</return_type>
<return_type>Array(Tuple(Int8, UInt64, Array(Float64), Array(Array(UUID))))</return_type>
<return_name>result</return_name>
<argument>
<type>UInt8</type>
Expand All @@ -58,17 +59,18 @@
<name>prop_vals</name>
</argument>
<argument>
<type>Array(Tuple(Nullable(Float64), UInt64, Array(Int8)))</type>
<type>Array(Tuple(Nullable(Float64), UUID, UInt64, Array(Int8)))</type>
<name>value</name>
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_cohort.py</command>
<command>aggregate_funnel</command>
<lifetime>600</lifetime>
</function>

<function>
<type>executable</type>
<name>aggregate_funnel_array</name>
<return_type>Array(Tuple(Int8, Array(String), Array(Float64)))</return_type>
<return_type>Array(Tuple(Int8, Array(String), Array(Float64), Array(Array(UUID))))</return_type>
<return_name>result</return_name>
<argument>
<type>UInt8</type>
Expand All @@ -91,11 +93,12 @@
<name>prop_vals</name>
</argument>
<argument>
<type>Array(Tuple(Nullable(Float64), Array(String), Array(Int8)))</type>
<type>Array(Tuple(Nullable(Float64), UUID, Array(String), Array(Int8)))</type>
<name>value</name>
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_array.py</command>
<command>aggregate_funnel</command>
<lifetime>600</lifetime>
</function>

<function>
Expand Down Expand Up @@ -124,11 +127,12 @@
<name>prop_vals</name>
</argument>
<argument>
<type>Array(Tuple(Nullable(Float64), Nullable(String), Array(Int8)))</type>
<type>Array(Tuple(Nullable(Float64), UUID, Nullable(String), Array(Int8)))</type>
<name>value</name>
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_test.py</command>
<lifetime>600</lifetime>
</function>

<function>
Expand Down Expand Up @@ -170,6 +174,7 @@
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_trends.py</command>
<lifetime>600</lifetime>
</function>

<function>
Expand Down Expand Up @@ -208,6 +213,7 @@
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_array_trends.py</command>
<lifetime>600</lifetime>
</function>

<function>
Expand Down Expand Up @@ -246,6 +252,7 @@
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_cohort_trends.py</command>
<lifetime>600</lifetime>
</function>

<function>
Expand Down Expand Up @@ -283,5 +290,6 @@
</argument>
<format>JSONEachRow</format>
<command>aggregate_funnel_array_trends_test.py</command>
<lifetime>600</lifetime>
</function>
</functions>
1 change: 1 addition & 0 deletions funnel-udf/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/target
152 changes: 152 additions & 0 deletions funnel-udf/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions funnel-udf/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[package]
name = "funnels"
version = "0.1.0"
edition = "2021"

[dependencies]
serde = { version = "1.0.104", features = ["derive"] }
serde_json = "1.0.48"
itertools = "0.11"
uuid = { version = "1.10.0", features = ["v4", "serde"] }


[profile.release]
lto = true
Loading
Loading