Home

Architecture

In a glow program, datasets are linked together by computation steps. Each dataset can be partitioned into dataset shards. Each computation step can also be partitioned into computation tasks.

Glow make full use of Golang's channel feature.

In local mode, data are fed into computation tasks by input channel(s), and output also via a channel to a new dataset.

In distributed mode, a group of tasks will run together in a server, pulling its own input dataset shards. Its output are streamed to local disk, which will be pulled by downstream tasks. But these piping work are not affecting the computation since the tasks only input and output via channels.

Architecture in distributed mode

Here are the components in distributed mode. Master, Agents, Driver, TaskGroup,

Glow Master

Master collects data from agents about resources.

Resources are CPU, memroy, allocated CPUs, allocated memory, etc.

Currently there is only a single master. But since master only has soft states from agents, we can easily extend it to a master cluster to avoid SPOF.

Glow Agent

Agent wear several hats:

1. reports system resources and usage to master.
2. accept tasks from driver program.
3. fetch the binary executable from driver program.
4. accept reads and writes for a dataset.

Driver program

Driver program is actually just the code that a developer will write. If executed with "-glow" option, it will drive the distributed execution.

1. create optimized execution plan, group tasks into task groups.
2. request resources from master.
3. allocate tasks to assigned servers.
4. if a dataset has input or output channels, write or receive from those datasets.
5. clean up intermediate datasets generated during run time.

TaskGroup program

Tasks usually can be grouped together. A taskgroup program actually also uses the same binary executable file as the driver program, but in task mode.

1. setup inputs and outputs for the tasks.
2. execute the tasks.

How executions are moved to remote servers?

One of the Golang's missing feature is the capability to move execution closure across the network, and not likely available any time soon. Given current situation, we can just move the whole binary code, but run in different modes, i.g., task mode and driver mode.

However, the implication is that the computation flow will be static. The flow graph can not be changed. One future way to allow dynamic flow is to pre-register all the flows, and dynamically choose one or several flows to run.

Examples:

Check this page first. https://github.com/chrislusf/glow_examples/tree/master/word_count

Contribution

Fork it, code it, and send pull requests. Better first discuss about the feature you want on the mailing list. https://groups.google.com/forum/#!forum/glow-user-discussion

License

http://www.apache.org/licenses/LICENSE-2.0