feat(web-analytics): Turn timezones into UTC offsets instead of timezone names #26409
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rafaeelaudibert
commented
Nov 25, 2024
| properties={"$session_id": session_id, "$pathname": f"/path{i}", "$timezone": distinct_id}, | ||
| ) | ||
|
|
||
| with self.assertRaisesRegex(Exception, "Cannot load time zone"): |
There was a problem hiding this comment.
Reasonable error?
robbie-c
reviewed
Nov 25, 2024
| @@ -1006,50 +1006,67 @@ def test_cohort_test_filters(self): | |||
| assert results == [["/path1", 1, 1]] | |||
|
|
|||
| def test_timezone_filter(self): | |||
| d1, s1 = "d1", str(uuid7("2024-07-30")) | |||
| d2, s2 = "d2", str(uuid7("2024-07-30")) | |||
| date = "2024-07-30" | |||
There was a problem hiding this comment.
For me to be suuuper confidant this works, I'd like to see some tests that cover before / after the daylight savings change for the same timezone
There was a problem hiding this comment.
Just added it, good call
rafaeelaudibert
commented
Nov 25, 2024
|
Size Change: 0 B Total Size: 1.16 MB ℹ️ View Unchanged
|
robbie-c
reviewed
Nov 26, 2024
| # Get the difference between the UNIX timestamp at UTC and the UNIX timestamp at the event's timezone | ||
| # Value is in milliseconds, turn it to hours, works even for fractional timezone offsets (I'm looking at you, Australia) | ||
| return parse_expr( | ||
| "if(isNull(properties.$timezone), NULL, (toUnixTimestamp64Milli(parseDateTimeBestEffort(assumeNotNull(toString(timestamp, properties.$timezone)))) - toUnixTimestamp64Milli(timestamp)) / 3600000)" |
There was a problem hiding this comment.
Let's add a test for key not present, value is NULL, and value is an empty string, I think you might need some extra logic to handle the empty string.
robbie-c
approved these changes
Nov 26, 2024
Timezone names on their own aren't that useful because they're basically a harder-to-use version of displaying events by city/region. Displaying them by offsets, however, can prove to be valuable. We're doing some magic to calculate the offset instead of simply using something like `timezoneOffset` because `toString` is one of the only functions in Clickhouse that allows us to use a dynamic timezone versus a hardcoded one. This might have some performance implications, we'll figure it out soon :). To workaround that, we're converting to a string, then parsing that back to a date, and getting the unix timestamp. Comparing the UNIX timestamp in a specific timezone versus the value at UTC allows us to get the offset at that day. See #26376
rafaeelaudibert
force-pushed
the
26376-turn-timezones-into-utc-offsets-instead
branch
from
4a43be8 to
d6361fd
Compare
rafaeelaudibert
deleted the
26376-turn-timezones-into-utc-offsets-instead
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Timezone names on their own aren't that useful because they're basically a harder-to-use version of displaying events by city/region. Displaying them by offsets, however, can prove to be valuable. We're doing some magic to calculate the offset instead of simply using something like
timezoneOffsetbecausetoStringis one of the only functions in Clickhouse that allows us to use a dynamic timezone versus a hardcoded one. This might have some performance implications, we'll figure it out soon :). To workaround that, we're converting to a string, then parsing that back to a date, and getting the unix timestamp. Comparing the UNIX timestamp in a specific timezone versus the value at UTC allows us to get the offset at that day.See #26376
Changes
Does this work well for both Cloud and self-hosted?
Yep
How did you test this code?
Comprehensive unit test + manual UI testing