Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add continuous aggregate refresh policies #49

Merged
merged 2 commits into from
Nov 7, 2023

Conversation

yuenmichelle1
Copy link
Collaborator

https://docs.timescale.com/api/latest/continuous-aggregates/add_continuous_aggregate_policy/

Starting_offset is 3 days which is smallest possible. If set to anything smaller, will get a refresh window too small error.
end_offset looks back to the past hour from when the refresh policy job runs.
Scheduling_interval is 1 day (so refresh policies will run daily)

For all continuous aggregates, all refresh policies are set to look back to the past 3 days through the past hour of when the refresh job is done to see if there are any changes, and will update the continuous aggregate. This job will be run daily.

db/schema.rb Show resolved Hide resolved
@yuenmichelle1 yuenmichelle1 requested a review from zwolf November 6, 2023 18:19
Copy link
Member

@zwolf zwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all refresh policies are set to look back to the past 3 days through the past hour of when the refresh job is done to see if there are any changes

I don't think I quite understand this, could you reword? And I think it might be worth putting the the explanation in the migration as a comment so that there's some immediate context in the repo w/o having to trace it back.

@yuenmichelle1
Copy link
Collaborator Author

yuenmichelle1 commented Nov 6, 2023

all refresh policies are set to look back to the past 3 days through the past hour of when the refresh job is done to see if there are any changes

I don't think I quite understand this, could you reword? And I think it might be worth putting the the explanation in the migration as a comment so that there's some immediate context in the repo w/o having to trace it back.

english is so hard.
All the refresh policies for the continuous aggregates that I have created will look back the past 3 days when refresh job is run. (Mainly because a 3 day lookback is the smallest window I can set for refresh policies).

As an example:

Refresh Continuous Aggregate Job is run today Nov 6th 1:47PM CST, the job will look at the classification_events table for the past 3 days (Nov 3rd) through the past hour (Nov 6th 12:47 PM CST), and will check for any changes in the classification_events table that have NOT been calculated in previous refresh jobs and then update the caggs with updated calculations.

@yuenmichelle1 yuenmichelle1 merged commit 20a7ec2 into main Nov 7, 2023
3 checks passed
@yuenmichelle1 yuenmichelle1 deleted the add-cagg-refresh-policies branch November 7, 2023 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants