Interfaces and shared infrastructure for generic task processing (also known as taskproc) at Yelp.
hello-world.py is a very simple annotated example that launches a task to echo hello world. From the root of the repository, run:
docker-compose -f examples/cluster/docker-compose.yaml \
run playground examples/hello-world.py
This will bring up a single master, single agent Mesos cluster using Docker Compose and launch a single task which will print "hello world" to the sandbox's stdout before terminating.
Other examples available include:
-
async.py Example of the async task runner.
-
dynamo_persistence.py Example that shows how task events may be persisted to DynamoDB using the
statefulplugin. -
file_persistence.py Example that shows how task events may be persisted to disk using the
statefulplugin. -
promise.py Example that shows how the promise/future task runner (not yet implemented) may be used.
-
subscription.py Example of the subscription task runner.
-
sync.py Brief example using the sync task runner.
-
timeout.py Example that shows how to timeout a task execution using the
timeoutplugin. -
retry.py Example that shows how to retry a task on failure using the
retryplugin. -
task_logging.py Example that shows how to fetch task logs from Mesos agents using the
loggingplugin.
From the root of the repository, run:
make
Plugins can be chained to create a task execution pipeline with more than one property. Please refer to persistence/retry/timeout examples.
Implements all required interfaces to talk to Mesos deployment. This plugin uses PyMesos to communicate with Mesos.
Implements an executor to timeout task execution.
Implements an executor to retry task execution upon failure.
Implements an executor to retrieve task logs from Mesos agents. Note that it has to be the immediate upstream executor of the mesos executor.
- authentication_principal Mesos principal
- credential_secret_file path to file containing Mesos secret
- mesos_address host:port to connect to Mesos cluster
- event_translator a fucntion that maps Mesos-specific events to
Eventobjects
TODO: documentation
Runners provide specific concurrency semantics and are supposed to be platform independent.
Running a task is a blocking operation. sync runners block until the running task has completed or a stop event is received.
Provide callbacks for different events in tasks' lifecycle. async runners allow tasks to specify one or more EventHandlers which consist of predicates and callbacks. Predicates are evaluated when an update is received from the task (e.g. that it has terminated and whether or not it has succeded) and if the predicate passes, the callback is called.
Running a task returns future object.
Provide a queue object and receive all events in there.