Source SQL plugin with REST Sink give poor performance

[manikandan-harman]

Hi,
I am using ekuiper 1.11 with SQL source and Rest sink plugin to send the data to cloud (Azure Event hub)

SQL Source Plugin: Getting 1500 rows every 2 second
Rule: SELECT * FROM SQLStream
SINK: REST sink configured with Azure event hub.

I am seeing that every row in the SQL converted into a message for REST sink. Meaning 1500 rows into 1500 REST API call.

I want 1500 messaged into a single message to REST API. Is this possible in SQL SOURCE Plugin?

I could see a lot of performance issue.

This is buffer configuration.

Am I using correct buffer size and other setting? Please guide us.

[ngjaying]

Hi @manikandan-harman ,

It seems the latency of project is the biggest, about 172ms. This means it can only handle 1000/172 ~ 5 message per second. If the data source has 750 qps consistently, there will be back pressure regardless how large the buffer size is. Do you know why the project op is so slow? It is related to your SQL actually.
Additionally, the REST sink has a latency of about 39ms, which is also much slower than the source.

It is a good idea to batch the outputs in this situation. You can use tumbling window like a 1 second window, to send out one message per second. Or using count window to send out based on message count. Also, feel free to open a feature request to support SQL source to ingest in batch. Thank you.

[manikandan-harman]

Able to achieve with count window function.
Thank you