| layout | title | categories | author | lang |
|---|---|---|---|---|
post |
Produce and consume Kafka messages over HTTPS with OVH Queue DBaaS |
queue |
sebgl |
en |
This tutorial assumes you already have an OVH account and have subscribed to the Queue DBaaS. If not, have a look at our Getting Started guide that will also introduce you to Kafka
Kafka protocol is built over TCP for multiple reasons, including performance and message ordering. However we realized that for some use cases, Kafka clients usage can be quite complex since many parameters have to be taken into consideration:
- Number of partitions and partition leaders
- Offset commit
- Consumer groups
According to your programming language of choice, Kafka clients can be more or less efficient and easy to use. Sometimes what you need is a simpler queue platform that you can use with any programming language. HTTP is the de-facto standard for that usage.
Queue DBaaS platform implements a simple HTTP layer that allows you to:
- Produce one or many messages to your topic with a POST request
- Consume one or many messages from your topic with a GET request
Moreover, it comes with HTTPS support, for free.
According to how you created your Queue DBaaS app, you chose one of the available region, which can be retrieved on your Sunrise Manager. Here we are interested in the HTTPS URL.
Example:
https://kafka.p1.sbg.queue.ovh.net
As with traditionnal Queue DBaaS usage with Kafka Clients, you will have to use:
- an SASL username (eg.
SASL_USERNAME=my-app.admin) - an SASL password (eg.
SASL_PASSWORD=aRT3u7R2TuRzhMmnLl) - a topic that your user is authorized to access (eg.
TOPIC=my-app.my-topic)
See our Getting Started and SASL guides for more information.
User authentication is made with a standard HTTP basic auth in which you must provide your SASL username and password:
$SASL_USERNAME:$SASL_PASSWORD
You can post one or multiple messages, using curl or HTTP clients in any language:
curl -u $SASL_USERNAME:$SASL_PASSWORD
-XPOST https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC
-d '[{"Value": "first message"},
{"Value":"second message"}]'
Please note in the example above that your message payload must be a JSON array of objects containing your message as a string in the "Value" field.
If this data model does not suit you, you can also post a message in raw format (accepts JSON, string, arrays, bytes...):
curl -u $SASL_USERNAME:$SASL_PASSWORD
-XPOST https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC?format=raw
-d '{"my": {"complex": ["json", "here"]}}'
You can retrieve one or multiple messages, using curl or HTTP clients in any language:
curl -u $SASL_USERNAME:$SASL_PASSWORD
https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC[?limit=2]
The optional limit parameter allows you to specify how many messages you want to retrieve.
Offsets are managed automatically at each GET request. It means that each message will be read only once. We plan to support manual offset and consumer group specification later on.
As seen in the examples above, messages are sent through a JSON object, and must be strings. This does not prevent you from using other data types, by encoding your binary data as base64 for example.
Despite the ease-of-use, using HTTPS instead of TCP implies a performance hit. To put it simple: if you need a very high read/write throughput, use a normal TCP-based Kafka Client.
As an example, here are numbers on producing and consuming "hello world" messages:
- via a TCP Kafka Client: 200k+ messages per second
- via the HTTPS endpoint: 5-10k messages per second
Note that the number of messages you send or receive with a single HTTP request has an impact on performance: don't hesitate to batch messages together.
Please feel free to contact us for any question, feedback, or improvement that may come to your mind !