The Concord Kinesis consumer allows Concord users to quickly and easily ingest messages from Kinesis queues into their Concord topologies. The project is, itself, open source and buildable, but can be used by simply fetching a prebuilt megajar and configuring a task deployment.
The latest release can be found at the following URL: https://storage.googleapis.com/concord-release/concord_kinesis_consumer-0.1.0.jar
To launch a Kinesis consumer, you'll need two things:
- A jar of a recent build of the Concord Kinesis consumer
- A run script / "shim"
- A json deployment manifest
The run script is responsible for defining the coniguration options. These include the following command line switches:
-topic
(required): The name of the topic to consume from Kinesis-ostreams
(required): A comma separated list of Concord streams on which we will emit every record pulled off Kinesis.-awskey
(required): Your AWS Access Key ID-awssecret
(required): Your AWS Secret Key-name
(required): The name of the computation (to register with Concord)-queuesize
(default: 16384): The maximum number of messages to buffer in memory
An example shim, runner.bash
might look like this:
#!/bin/bash
JAR="/path/to/concord-kinesis.jar"
AWSKEY="..."
AWSSECRET="..."
exec java -jar $JAR -topic my-topic -ostreams foo,bar\
-awskey $AWSKEY -awssecret $AWSSECRET -name kinesis-consumer
An example JSON manifest for the above configuration might look like this:
{
"zookeeper_hosts": "localhost:2181",
"zookeeper_path": "/concord",
"executable_name": "runner.bash",
"compress_files": ["concord_kinesis_consumer.jar", "runner.bash"],
"computation_name": "kinesis-consumer",
"mem": 1024,
"cpus": 1.0,
"instances": 1
}
For this example, a simple flat directory structure will suffice:
kinesis-example/
deploy.json
runner.bash
concord_kinesis_consumer.jar
To launch, simply run:
$ concord deploy deploy.json
and look at the Mesos dashboard. You should see the computation running!