Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Batch Publishing #36

Open
escb005 opened this issue Nov 6, 2023 · 6 comments
Open

[Feature Request]: Batch Publishing #36

escb005 opened this issue Nov 6, 2023 · 6 comments

Comments

@escb005
Copy link
Contributor

escb005 commented Nov 6, 2023

Using publisher confirms, sending multiple messages currently takes a lot of time since it waits for the confirmation between the individual sending.

A function that first publishes a list of messages and then waits for the confirmation would circumvent this issue.

@jxsl13
Copy link
Owner

jxsl13 commented Nov 6, 2023

@jxsl13
Copy link
Owner

jxsl13 commented Nov 6, 2023

Possible but may lead to potential out of order messages or to duplicated messages when a whole batch is republished.

Other solutions may include having a data format consisting of a list of messages which are correctly ordered but just need more memory for processing.

@escb005
Copy link
Contributor Author

escb005 commented Nov 6, 2023

Would this still work if the messages are routed to different queues or even different exchanges?

@jxsl13
Copy link
Owner

jxsl13 commented Nov 6, 2023

A RabbiqMQ message can be up to like 2G+ big. So the big blob of multiple messages would be routed as a whole.

I do highly NOT suggest to have messages that big, it's just that you can have like somewhat bigger messages like a few 100KiB.

@jxsl13
Copy link
Owner

jxsl13 commented Nov 6, 2023

you'd have to preprocess your messages somewhat if possible so that each list is routed to the same destination and a single list only contains homogenous mesages.

@jxsl13
Copy link
Owner

jxsl13 commented Mar 26, 2024

wrapping the batch in a transaction may help but needs further investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants