Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve clickhouse table creation #154

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Conversation

Felixoid
Copy link
Collaborator

I need as well to test all the tables for performance before merging it

@Felixoid Felixoid force-pushed the optimized-create-clauses branch from 854d70e to c2d650a Compare October 31, 2024 13:44
Copy link
Contributor

@Hipska Hipska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, but I'm only wondering what exactly you mean with "will be almost always 0"?

README.md Outdated Show resolved Hide resolved
Co-authored-by: Thomas Casteleyn <[email protected]>
@Felixoid
Copy link
Collaborator Author

Felixoid commented Nov 5, 2024

DoubleDelta takes the second differential for the value.

E.g., when we have a column with values 60, 120, 240, 300, ...., the values stored in the file will be 60, 60, 0, 0..., then compressed. It's the best codec for monotonic values.

@Felixoid
Copy link
Collaborator Author

Felixoid commented Nov 8, 2024

I still haven't tested the performance of this changes. Is a bit stuck on go-graphite/graphite-clickhouse-tldr#6 to have the tags and all other stuff in the DB

@Hipska
Copy link
Contributor

Hipska commented Nov 12, 2024

Thanks to inform this, I was going to apply this to PROD, but will wait. It is doing fine on a small DEV server:

    ┌─table─────────────────┬─column────┬─type────┬──────rows─┬─disk───────┬─avg_size─┬─compressed─┬─uncompressed─┬─────compress_ratio─┐
 1. │ default.data_backup   │ Path      │ String  │ 133438903 │ 56.81 MiB  │ 0.45 B   │ 56.78 MiB  │ 12.43 GiB    │  224.1160238907609 │
 2. │ default.data_backup   │ Value     │ Float64 │ 133438903 │ 86.55 MiB  │ 0.68 B   │ 86.51 MiB  │ 1018.06 MiB  │ 11.767609582355778 │
 3. │ default.data_backup   │ Time      │ UInt32  │ 133438903 │ 231.96 MiB │ 1.82 B   │ 231.92 MiB │ 509.03 MiB   │  2.194811239521079 │
 4. │ default.data_backup   │ Date      │ Date    │ 133438903 │ 2.32 MiB   │ 0.02 B   │ 2.29 MiB   │ 254.51 MiB   │ 111.14184778172574 │
 5. │ default.data_backup   │ Timestamp │ UInt32  │ 133438903 │ 24.22 KiB  │ 0.00 B   │ 2.90 KiB   │ 145.41 KiB   │  50.21787521079258 │
 6. │ default.graphite_data │ Path      │ String  │ 148363663 │ 7.51 MiB   │ 0.05 B   │ 7.47 MiB   │ 13.70 GiB    │ 1877.6107315995705 │
 7. │ default.graphite_data │ Value     │ Float64 │ 148363663 │ 61.22 MiB  │ 0.43 B   │ 61.19 MiB  │ 1.11 GiB     │ 18.498867365596805 │
 8. │ default.graphite_data │ Time      │ UInt32  │ 148363663 │ 15.66 MiB  │ 0.11 B   │ 15.63 MiB  │ 565.95 MiB   │ 36.213080019697834 │
 9. │ default.graphite_data │ Date      │ Date    │ 148363663 │ 752.65 KiB │ 0.01 B   │ 725.17 KiB │ 282.98 MiB   │ 399.58597313402686 │
10. │ default.graphite_data │ Timestamp │ UInt32  │ 129609900 │ 20.64 KiB  │ 0.00 B   │ 2.04 KiB   │ 141.25 KiB   │  69.27155172413794 │
    └───────────────────────┴───────────┴─────────┴───────────┴────────────┴──────────┴────────────┴──────────────┴────────────────────┘

Copy link
Contributor

@Hipska Hipska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use TTL based on Date column to align with graphite data table? That way it also won't cause issues when Version isn't representing a timestamp anymore..

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
Co-authored-by: Thomas Casteleyn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants