Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redesign filtered data reporting #154

Open
colmsnowplow opened this issue Jun 24, 2022 · 0 comments
Open

Redesign filtered data reporting #154

colmsnowplow opened this issue Jun 24, 2022 · 0 comments
Labels
bug Something isn't working refactor Suggestion of a better way to factor the code

Comments

@colmsnowplow
Copy link
Collaborator

colmsnowplow commented Jun 24, 2022

Two issues:

In transform.go, we don't report timeTransformed for filtered messages.

The reason for this was to avoid misreporting unfiltered data as faster than it actually was, by combining it with filtered data. However this was a mistake, since filtered data is handled separately anyway, so there's no risk of that.

Additionally, we report the timestamp for filtered messages only at the end of the process, when we create a filterResult.

This means that our reporting of transform latency is defined as the time between message pulled and the completion of that event's transformation, but the filter latency is the delta between message pulled and the completion of the batches transformations.

We should change this so that those two timestamps represent similar things (filters are transformations under our current mode), and also rethink the design of this reporting, and consider including filtered messages in the TargetWriteResult rather than separately from that.

@colmsnowplow colmsnowplow added bug Something isn't working refactor Suggestion of a better way to factor the code labels Jun 24, 2022
@colmsnowplow colmsnowplow changed the title Report filter latency more accurately Redesign filtered data reporting Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working refactor Suggestion of a better way to factor the code
Projects
None yet
Development

No branches or pull requests

1 participant