Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task start time conditions #105

Open
eric opened this issue Jul 28, 2014 · 9 comments
Open

Task start time conditions #105

eric opened this issue Jul 28, 2014 · 9 comments

Comments

@eric
Copy link
Contributor

eric commented Jul 28, 2014

Tasks should be able to have a specified time that they are not started until. Tasks will not be eligible to claim until the time has passed.

@eric
Copy link
Contributor Author

eric commented Jul 30, 2014

@aphyr: what format would you suggest specifying time in and what call should we use to get the current time?

@aphyr
Copy link
Contributor

aphyr commented Jul 30, 2014

Uh, this opens up a whole nest of issues, including that people will think the option actually means something. I'd avoid implementing this if at all possible.

@eric
Copy link
Contributor Author

eric commented Jul 30, 2014

I've ran into all sorts of reasons why I've wanted to put off running a task for some period of time.

What about something that says, "This can't be claimed until X seconds has passed"?

It would allow for doing backoff, but wouldn't be tied to a client clock.

@aphyr
Copy link
Contributor

aphyr commented Jul 30, 2014

Skuld's design assumptions allow for clock skew on the order of minutes to hours. Second-scale timeouts are almost definitely not feasible.

@eric
Copy link
Contributor Author

eric commented Jul 30, 2014

The problem I have that I was hoping to solve with this is cases where I look at a task and see that some other resource that this task depends on isn't available right now, so I don't need to try to run this task again for another X minutes.

As it is with a simple message queue (like what I use today), my only option is to throw it back into the end of the queue, which means that it will continue to see the same tasks over and over again while they're requeued until such a time that the other resource is ready again.

@aphyr
Copy link
Contributor

aphyr commented Jul 30, 2014

You can use linear-time as a realllly loose proxy for "global time", but it can and will drift by huge volumes, and has no relation to anything outside Skuld. Heck, it may not even be close to other nodes.

@aphyr
Copy link
Contributor

aphyr commented Jul 30, 2014

The correct solution is task dependencies. There's a writeup of this stuff in the Factual wiki; not sure if I ported it to the github repo.

@aphyr
Copy link
Contributor

aphyr commented Jul 30, 2014

Oh, or, if your consumer is the one that discovers the dependent service is down, just have the consumer stop processing records for a bit.

@eric
Copy link
Contributor Author

eric commented Jul 30, 2014

I see #15 that mentions dependencies, but I was assuming that just meant that one task has to complete before another one does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants