Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some issues with benchmark tool #2

Open
kozlovic opened this issue Sep 27, 2019 · 3 comments
Open

Some issues with benchmark tool #2

kozlovic opened this issue Sep 27, 2019 · 3 comments

Comments

@kozlovic
Copy link

Hello, I work on the nats.io team and was made aware of your project.
Thank you for integrating with NATS!

I noticed very low numbers for the NATS report and started to dig into it.
What I have found first is that you can't run the benchmark without first fixing the import:

https://github.com/ThreeDotsLabs/watermill-benchmark/blob/master/cmd/main.go#L10
Should be: "github.com/ThreeDotsLabs/watermill-benchmark/pkg"

Then I tracked down the biggest part of the slowness in the NATS consumer side to the unmarshaling (https://github.com/ThreeDotsLabs/watermill-nats/blob/master/pkg/nats/marshaler.go#L41). For 100,000 messages it is taking 2secs on my machine. This is something to look into.

I would recommend that you set the AckWait to higher number than 1sec, conversely, the default CloseTimeout is too high IMO.

You don't expose it seems the start position for the subscription. So I am not sure if this is why you stop/start the subscriptions at initialization time. With NATS Streaming you can pass options to specify where to start in the stream (DeliverAllAvailable() for all messages, etc..).

I was surprised with Kafka numbers compared to NATS. Should not the Kafka publisher require acks from the cluster? (https://github.com/ThreeDotsLabs/watermill-kafka/blob/master/pkg/kafka/publisher.go#L80) something like:

config.Producer.RequiredAcks = sarama.WaitForAll

Thanks for your attention and effort!

@kozlovic
Copy link
Author

kozlovic commented Sep 27, 2019

Also, upgrading the streaming server to latest (0.16.2) will help with file store performance on read.
But overall, there is a significant overhead with this lib compared to bare NATS streaming API. That is, whatever consumer performance you get, if you start a consumer with say the bench tool that we have (https://github.com/nats-io/stan.go/tree/master/examples/stan-bench), you should see a huge difference (even if I bypass the marshal/unmarshal in your nats implementation).

go run examples/stan-bench/main.go -np 1 -ns 0 -n 1000000 benchmark
go run examples/stan-bench/main.go -np 0 -ns 1 -n 1000000 benchmark

(this will send all first, then consume, similar to what your benchmark tool does).

@m110
Copy link
Member

m110 commented Oct 12, 2019

Hi @kozlovic! I'm sorry for such late response, I didn't receive notification of this issue for some reason.

Thanks a lot for pointing out possible issues, we will definitely look into it! I'll keep you updated.

@m110
Copy link
Member

m110 commented Oct 12, 2019

Import paths fixed in #4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants