Processing api calls with background processing

[oliver-kriska]

Hi, I have just quick idea/question/brainstorm. Why don't you process data from api calls with "background jobs"? I don't see your internal performance stats from AppSignal but this approach can really speedup JS api calls. You can use full power of Oban if your posgresql has some free resources. Or maybe better solution would be use GenStage to process them in bulks. You can even create own processor for every page with DynamicSupervisor. You don't have to care about queue persistence because you don't have sensitive data, just stats. But that can be handled with rolling update in your deployment strategy and some small optimization.

BTW: don't do this in controller
https://github.com/plausible/analytics/blob/master/lib/plausible_web/controllers/api/external_controller.ex#L67-L125

[ukutaht]

I'm 100% sure we will evolve to a queue architecture with async processing as opposed to the current synchronous processing approach. The main benefit I see is reliability/uptime improvements during peak load times. Instead of crashing under heavy load, we could just fill up a queue and show a little notice on the dashboard when the queue is large enough to cause a visible delay in processing.

So far we've managed without this, although as you can see on our status page we are susceptible to crashing under heavy load.

Adding an intermediate interface to the pipeline will slow down development of new features though. I'm planning on this once we're a bit more stable with the product itself.

[ukutaht]

Yeah I agree API response time will basically be instant if the only op is adding to a queue. We could probably achieve a similar result by invoking Task.async on each incoming pageview.

I don't see why it would reduce our resource usage. Introducing a queue does not change the need to process incoming pageviews. If anything I'd expect memory consumption to increase to accommodate the queue. How do you figure it would reduce resource usage? Where would the savings on memory and/or CPU cycles come from?

[oliver-kriska]

Let's say we are talking about GenServer. You have 1000 calls at same time (or some small period). Now you are doing 1000 inserts and calls via db connection worker. Same with Task async and even a little bit more because of 1000 new processes. With GenServer you can call 1 bulk insert in one db connection worker. So it maybe you can have small increase of memory but not a lot, like 0-10 MB. But will save some resources from clickhouse and also phoenix.

[oliver-kriska]

Sorry, GenStage, not GenServer. :-)

[ukutaht]

Batching inserts is a great optimization which is already implemented. See https://github.com/plausible/analytics/blob/master/lib/plausible/event/write_buffer.ex

[oliver-kriska]

great. So looks like you have 30% of GenStage implemented by yourself :)