Logpipe is a utility for managing a batch processing pipeline.
With repeated execution via cron, files flow through the following 4-stages:
stage1-incoming - incoming raw files
stage2-holding - candidates awaiting validation for stage3
stage3-processing - raw files awaiting processing
stage4-outgoing - cooked files awaiting transport to S3
Actions executed in the processing stage can be written in any language and are executed using GNU parallel.
The final stage of the pipeline uploads the processed (cooked) output to Amazon S3 via s3cmd.
Logpipe is written in Bash and makes use of non-portable shell features.
There is an intentional lack of error handling in logpipe. Unexpected ouput will generate an email from the cron daemon.
Actions are executable programs called directly by logpipe via GNU parallel.
Individual files in stage3 are distributed as command line arguments to processes created by GNU parallel. The number of concurrent jobs will scale to the number of available CPU cores. Although unimplemented, a minor modification allows distribution of jobs to networked machines accessible via ssh.
Actions will be executed in natural sort order against the files in stage3.
Output files should be created in the path pointed to by the STAGE4 environment
variable which is inherited by action processes. Global variables are defined
in environment.sh
A final action, 99delete removes files from stage3 and should be the last
action that is executed. It can be referred to as a template for writing your
own actions but should be left in place to ensure files are removed properly.
Actions should depend on this behavior and not remove or otherwise modify files
themselves.
Logpipe can be called in one of four ways:
Usage: logpipe [options]
Options:
-h, --help Display this help and exit
-m, --move Prepare raw files for processing
-p, --process Execute actions
-u, --upload Upload cooked files to Amazon S3
An example schedule consists of executing logpipe via cron every 10 minutes:
*/10 * * * * /path/to/logpipe --move
*/20 * * * * /path/to/logpipe --process
*/30 * * * * /path/to/logpipe --upload
sudo apt-get install s3cmd parallel coreutils
The default installation path is /usr/local/ but you may override this by
defining PREFIX at install time. At a minimum, you must define S3_URI and
STAGE_PREFIX to INSTALL.sh.
sudo \
PREFIX=/custom/path \
S3_URI=s3://your_bucket/ \
STAGE_PREFIX=/srv/logpipe \
./INSTALL.sh