Performance on Heroku

[maxsz]

We recently started running rapns on heroku. Our goal is to send bursts of a few hundred thousand (possibly even more in the future) push messages per minute. Unfortunately the performance of our setup on heroku is really bad (like a few hundred pushes per minute). So I was thinking what would be the best option to improve the performance while still having the convenience of using heroku servers. Basically I think there are two options:

1. Improve the rapns push-loop performance
2. Support horizontal scaling

What are your thoughts on this?

[mattconnolly]

Horizontal scaling is difficult when the messages to be sent are being stored in a database. You may find that the bottleneck is actually how fast Rapns can pull messages from the database.

On the other hand, storing them in the database has a very low cost of entry and is an excellent solution for scheduled (deferred) message delivery.

I've been investigating using ZeroMQ as a messaging layer which would allow far better horizontal scaling and delivery of messages directly without needing them to be stored in the database at all (think email delivery). I would make this an addition to the existing database method.

My use case for rapns is very low volume at the moment, so this way down my list of things to do.

[ileitch]

How many connections do you have? Have you disabled error checking? With many connections you will be constrained by the global internal lock of MRI, JRuby or Rubinius may help you achieve higher throughput.

But ultimately, support for horizontal scalability would be great.

You could also try implement a ZeroMQ (or other) backend yourself, see here: https://github.com/ileitch/rapns/wiki/Implementing-your-own-storage-backend

[ileitch]

Any progress on this?

[maxsz]

Unfortunately I haven't had the time to investigate this further, yet.

@mattconnolly I don't think that on Heroku it is the speed of the database but rather the slow performing rapns push-loop. After all the performance of one dyno is quite bad compared to a real machine.

As there is no way to scale vertically on Heroku (apart from doubling the memory), I don't think that it makes sense to spend time on improving the performance of rapns itself, unless we could pull off a very significant improvement :)

@ileitch I haven't played with the number of connections, so this is the default of one. Error checking is disabled.

I'll let you know, when there is any progress from my side.