Performance Issues with Postal 3.3.4 on 16 CPU / 64GB RAM Server

[tanaha2002]

Hi,

I'm experiencing significant performance issues with Postal 3.3.4 on a dedicated server with 16 CPUs and 64GB of RAM. I'm seeing a maximum throughput of around 75 messages/minute, which is far too slow. Sending a newsletter to under 200k subscribers has taken over a week and is still not complete.

I've tried the configurations suggested in #697 and #1907 , but haven't seen any improvement.

postal.yml

workers:
  quantity: 50
  threads: 60

main_db:
  host: 127.0.0.1
  username: xxxx
  password: xxxxxx
  database: postal
  pool_size: 150

mysql

innodb_buffer_pool_size	= 3048M
max_connections		= 9000
connect_timeout		= 6
wait_timeout		= 31536000
max_allowed_packet	= 256M

I understand that the quantity and threads settings in postal.yml are deprecated. I've also tried starting Postal with the --scaleoption:
postal start --scale worker=50

After all my Postal still running very slow, it can't even read 200 messages/minute. And now it just still 75 messages/minute is max. I were sent a newsletter and it took me 1 week and still not done ( < 200k customer).

However, the performance remains very slow.

My server is hosted by Wholesale Internet.

Could you please advise on any further steps I can take to improve Postal performance?

My server metric currently when postal running

[willpower232]

Are you running MySQL on a separate server? I presume you restarted MySQL and Postal after changing the settings?

[tanaha2002]

Yes that sounds right, and the screenshot you added elsewhere shows workers are doing things.
I wonder what would happen if you reduced the worker settings in the config file back to their original values but kept the scaled 50

I just did it and start another campaign, let wait, I will provide the result tomorrow.

[tanaha2002]

So after try uninstall everything and fresh re-install and start postal with --scale to 50 worker. Now i can reach 228.82 messages/minute. So i don't even know what cause the problem before while the 2nd install is same with 1st. But i can reach 228.82 messages/mintue now. No config anything in postal.yml just config in my.cnf of MariaDB and --scale command. But 228 is still low for this hardware spec right?

[willpower232]

Interesting, I guess one of the changes didn't quite land properly somehow.

There isn't really any way of knowing how much a server can handle off hand. Presumably your htop is much busier now?

[tanaha2002]

Interesting, I guess one of the changes didn't quite land properly somehow.

There isn't really any way of knowing how much a server can handle off hand. Presumably your htop is much busier now?

hmm, after period of time watching it. It's same with before. Still only consume ~12gb RAM and CPU didn't use too much, just like 3 of them is 30%.

[willpower232]

well I guess you could add some more workers and see what happens then

[UEAB-BL]

Maybe not super helpful but if you check the logs by running postal log (if i remember correctly), you can see if the workload is being shared or if its just the same worker doing all the work.

[tanaha2002]

Maybe not super helpful but if you check the logs by running postal log (if i remember correctly), you can see if the workload is being shared or if its just the same worker doing all the work.

Yes, all of them is working, but it cannot reach 200 messages/min

[Almeonamy]

We have the same issue. And I think the bottleneck is in the handling of http-requests. The response time of Postal's API is approximately 3-7 seconds and is grows throughout the day. I think the problem is in raw message saving process. I noticed that the size of the raw's table is greater than 2GB.

The chart shows response time in ms. The decrease in the center is due to server restart.

[tanaha2002]

We have the same issue. And I think the bottleneck is in the handling of http-requests. The response time of Postal's API is approximately 3-7 seconds and is grows throughout the day. I think the problem is in raw message saving process. I noticed that the size of the raw's table is greater than 2GB.

The chart shows response time in ms. The decrease in the center is due to server restart.

Well, nice analysis. Do you have any ideas to solve this bottleneck?

[Almeonamy]

Perhaps I could try creating multiple "virtual servers" in the Postal. Each "server" would have its own database and table for storing raw messages. However, I haven't tried this yet.

[willpower232]

Interesting, also sounds like #3172

Presumably a database index is missing or not being used in the raw tables so they get slower to use with time which gets particularly noticeable with more and/or bigger messages.