What are you working on?

[Miksus]

Hi all!

It would be nice to know how people are using the library. It would be nice to hear how the library can be used. I'm also looking for inspiration for example projects so that I could demonstrate the usage of the library in practice. Reading (and writing) technical documentation is not always fun.

Feel free to share your project here.

[ayr-ton]

Mainly for scheduling ETLs and creating data pipelines.
My first implementation was with Airflow and Rocketry is working well as a replacement.

[ayr-ton]

Right now I'm using just one worker, so it's working as intended. But probably this could fail with more workers as they don't know each other sessions. Are you planning the API to also coordinate events between tasks of different workers? Or do you picture this coordination differently?

[Miksus]

I suppose you mean by one worker that you have one instance of Rocketry running. Currently, one could run multiple workers if they force the logs to be read every time statuses are inspected (there is a config) and the workers use the same log repository. However, there is a considerable risk that two workers start the same task: worker A inspects whether the task can be started --> worker B inspects whether the task can be started --> worker A and B both start the task.

To overcome this, I have been thinking of having a repository lock. Basically a class with a similar interface as threading.Lock but this would inspect a repository (an SQL database, a REST API, MongoDB etc.) whether the task is locked or not. The lock in this case tells Rocketry that something else is inspecting the task and it should not touch it at that moment. Then one could specify whether a threading lock is sufficient or where they want to keep the locks. This solution also requires only a little work on the system but a bit to Red Bird.

Would be also a nice addition to Red Bird. What do you think about this idea?

[Miksus]

Also, I think there should be a way to store the outputs of tasks in some sort of datastores if arguments are pipelined but this I haven't yet figured out as the outputs can be any Python objects that may not fit typical data stores (ie. as rows to SQL).

[ayr-ton]

This is an excellent discussion, I like it.
I didn't know the possibility of forcing the same logs on different workers and the feature it brings.
Right now what I'm doing is creating a simple queue implementation in Redis for saving data that I need to be shared across different tasks.
I had broken down a worker into two others separating tasks (I don't have the same task in the two different workers). Some tasks are triggered more often and then I check what is in Redis and do something or nothing if the queue is empty.
One possibility would be storing the log in Redis itself, but I need to learn more to make it. This could be a good test to define a storage interface for other repositories.

I'm also learning about the Dataflow architecture structure that Apache Beam implements and the in-memory PCollection (this is a separate topic from the above one):
https://www.educative.io/answers/what-is-pcollection-in-apache-beam
The idea behind mimicking it is to have an in-memory partition in Redis, let's say with a month of information extracted from different tasks, archiving this in object storage and other tasks can pick this blob that looks like a PCollection and loading in BigQuery, then hashing to mark the data. I didn't like Apache Beam in the beginning because the pipeline syntax is very cluttered and I don't have scheduling available. But the data structure is powerful.

The full abstract that Apache Beam uses as reference architecture:
http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
Also:
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

Basically, I'm trying to achieve a low I/O and memory mimicking of Apache Beam, with scheduling via Rocketry, Redis, and cloud buckets.

[ayr-ton]

P.S.: Actually Redbird seems a good candidate for making pythonic PCollections, it could support the same methods in a way that doesn't remember java Apache Beam so much.

[thp44]

Making a repo to centralize varios small tasks to replace 'windows task scheduler'

• checking and posting license usage
• uploading user photos to an api
• uploading project metadata to a search engine
• retrieving data from google services
• ms graph api tasks/crawling
• crawling our fileshare
• etc...