Scheduling a task in a Distributed Environment like Kuberneted

[ibiswas]

I have a FAST API based Rest Application where I need to have some scheduled tasks that needs to run every 30 mins. This application will run on Kubernetes as such the number of instances are not fixed. I want the Scheduled Jobs to only trigger from one of the available instance and not from all the running instances creating a Race condition, as such I need some kind of locking mechanism that will prevent the schedulers to fire if one is already running. My App does connect to a MySql compatible Aurora DB running on AWS. Can I achieve this with ApScheduler, if not are there any alternatives available?

[agronholm]

This is possible with the upcoming v4.0 version which implements shareable job stores and message hubs. The first alpha release is very close. Please follow #465 for updates.

[agronholm]

It is, but since you are not running PostgreSQL, you will also need to deploy a messaging service separately. Redis and MQTT are supported out of the box. I am in the process of writing the instructions for setting up all of this, in addition to making changes to the synchronous API for WSGI compatibility.

[ibiswas]

So I was able to use APScheduler with Aurora DB using the SqlAlchemyJobStore as AuroraDB is MySQL compatible, will I still need Redis or MQTT when the shared JobStore feature arrives for v4, my idea was to set the number of instances to 1 and the shared Job should take care of the rest. Am I missing something?

[agronholm]

Are you talking about APScheduler 3 or 4? APScheduler 3 has no way for the schedulers to coordinate with each other so it's possible for jobs to get executed twice or not at all, depending on the situation. The recommended workaround has been to limit the number of schedulers to 1 per job store.

[ibiswas]

Hi Alex can you please confirm if the previous statement is accurate.

[ibiswas]

Sorry didn't refresh my screen, looks like u did respond. I meant with v4 does the below statement seems accurate

So I was able to use APScheduler with Aurora DB using the SqlAlchemyJobStore as AuroraDB is MySQL compatible, will I still need Redis or MQTT when the shared JobStore feature arrives for v4, my idea was to set the number of instances to 1 and the shared Job should take care of the rest. Am I missing something?