Task Development

Hello world task

The following snippet will show you how to create the simplest of the tasks.

from neuroglancer.pipeline.taskqueue import TaskQueue, RegisteredTask

class ExampleTask(RegisteredTask)
    def __init__(self, message):
        super(ExampleTask, self).__init__(message)
        self.message = message

    def execute(self):
        print self.message

if __name__ == '__main__':
    queue = TaskQueue()
    task = ExampleTask("Hello World")
    queue.insert(task)

Your task will be pushed to a google cloud task, a container will get your task and run execute printing your message.

Under the hood.

Your task will be first serialize

>>> print task.serialize()
{'message': 'Hello World', 'class': 'ExampleTask'}

This json string will contain all the arguments to the constructor of your class.

A worker will receive this string and thanks to some meta-programming magic run: task = ExampleTask("Hello World")

Now the task is ready to be executed by the worker.

try:
  task = queue.lease()
  task.execute()
  logging.info("success") # logging is sent to google cloud logging
except:
  logging.error() # send stack trace and other information to google cloud logging

I don't understand how does the worker know what to execute

Forgot to mention, you have to push your new task to github. travis-ci will get your new commit and build a container image out of it. We can later use this image to spawn workers.