Postal 3.x - Cluster

[tpnsolutions]

I know this has been asked for previous versions of Postal, so in a way I want to clarify things for 3.x

My goal is to build a cluster of VPS nodes which each handle a portion of mail processing for a few key purposes.

1. Load Balancing
2. Rate Limiting
3. Blacklist Management

== Load Balancing ==
We have a bunch of mailing list servers amongst others which rely on sending lots of mail (in the hundreds of thousands to millions) per day. Distributing the load across a cluster of servers allows for more mail to be sent out faster. Also allows us to conduct maintenance on one server while the load gets sent to another node therefore reducing downtime seen by our customers.

== Rate Limiting ==
We acknowledge that too much mail coming from a single IP address, even when there are multiple lists from different clients is seen by many providers as SPAM. Therefore we distribute when possible mail across multiple nodes to increase the rate at which we can deliver to these providers such as Gmail, Microsoft, etc.

== Blacklist Management ==
Because of the above "Rate Limit" issue, along with other false positives, we like to build a cluster that can be used to distribute mail through a node that is capable of reaching the destination server when we've been temporarily blacklisted, while we work with the provider to either get whitelisted or off their blacklist.

== Changes in 3.x ==
I noticed a few big changes with 3.x, in particular that it no longer relies on RabbitMQ, or the "cron" feature of the previous version.

Is it to be assumed that, simply setting up a centralized MariaDB server, then pointing each node at this server along with per-node configurations (DNS related primarily) is all that is needed to accomplish the cluster I am after?

In the past I was instructed to only setup "one" node which had the "cron" and "RabbitMQ" instance running on it... How does 3.x address these tasks given the "cron" and "RabbitMQ" requirements have been removed?

== Internal Routing ==
From the internal routing perspective, "outbound" mail from any of our "Service" nodes to "Postal" would be handled by software on those nodes by way of a "transport map" which utilizes a custom script that checks the health of each of the "Postal" nodes (deliverability per domain). Our system checks the "Postal" nodes to see whether they are able to connect to a given destination provider, then whitelist the successful ones. Finally relaying outbound traffic from the "Service" node to one of the "Postal" nodes for delivery.

"Inbound" traffic would be done in a similar manner, whereby a message would get received at one of the "Postal" nodes by way of MX routing, then forwarded onto the appropriate "Service" node.

Anyways, sorry for the long thread, would appreciate your feedback on this!

*** We're also looking at deploying a "master-master" cluster of MariaDB servers to add further fault tolerance and performance improvements ***

[rbustos]

We run LoadBalancer + 4 postal nodes (v2) + remote rabbit + xtraDB cluster (3 nodes). It will be interesting how they decide which task they should take without Rabbit.

[TimE90]

Also very interested about this way of setup.

[adamcooke]

Is it to be assumed that, simply setting up a centralized MariaDB server, then pointing each node at this server along with per-node configurations (DNS related primarily) is all that is needed to accomplish the cluster I am after?

In the past I was instructed to only setup "one" node which had the "cron" and "RabbitMQ" instance running on it... How does 3.x address these tasks given the "cron" and "RabbitMQ" requirements have been removed?

That is correct. The workers will elect a leader which will handle the actions which were previously handled by the cron process. If the elected worker disappears a new worker will take over that role within 5 minutes.

All state and queueing is now handled only within the database.

There was a lot of other information in your message but this seemed to be the only actual question. Did you have any specific areas you wanted advice or feedback on?

[tpnsolutions]

Thanks for the reply...

I guess I'm wondering "which node" will handle the "outbound" request?

If say I send SMTP traffic to "node #1", will that be the node that sends the outbound message? Or would perhaps "node #3" end up delivering the message? How does the system know which node to handle the traffic?

The reason is, we are distributing messages amongst other reason to a node that is capable (based on an internal test) of sending messages to the destination server.

I guess in summary I'm looking for a bit more intel regarding how queuing is handled, and decisions are made regarding who (which node) is handling what in a cluster environment. If it's the database that does this directly or indirectly what information is added to the database to aid in this decision?

Heh heh, sorry if I've confused the topic. I'm just trying to make an informed decision in my own architecture design as we CANNOT add more than one IP per node at this time.

[adamcooke]

When a message is received by Postal (either one which is incoming or one which should be sent to another server) it will be placed into a queue. The queue is monitored by all workers and any worker can take a message from the queue.

There is a caveat though. If you use the IP pool functionality, messages will be assigned an IP address when they are added to the queue and then only workers on a host which actually has the IP assigned will de-queue that message.

[JohnHammell]

@adamcooke , sorry to bring back this thread from 6+ months ago.

I have similar concerns with sending outbound through a node & the queue sending that message through a different node with a different IP address. For the same, or maybe similar, reason as @tpnsolutions in order to help address the issue of IP rate limiting by email providers when sending a large amount of email.

Additionally, just like @tpnsolutions I also can use only a single IP address per node.

Question: Is it possible to use the IP pool functionality of Postal with only a single IP address for each node in order so that messages sent through each node are assigned the IP address of that node?

That way we could be assured that the node# we intended that email to be sent out through is the node that the queue also sends it through, rather than through a different node.

[adamcooke]

I'm not sure on your use case but if you want to ensure that a specific node is the one which initiates the outbound connection to an external SMTP server, you can use IP pools with one IP in each.

Just remember, if a node isn't running, the emails which are assigned to its IP pool will not be removed from the queue and won't send.

[tpnsolutions]

@adamcooke,

What I've done "manually" within my own email cluster (not yet using postal) is:

1. setup a scheduled collection of all unique domains we send to
2. test each domain to against each of our nodes to see if we theoretically should be able to send to them (the telnet way looking for obvious errors)
3. if a node can successfully connect, we create a "A Record" on our DNS server to to the tune of "domain.relay.ourdomain.com" with the IP address of the successful node.
4. if unsuccessful and a record exists, we delete it
5. on the server(s) that are sending "using" the cluster, we setup transport maps for known domains mapped to their "domain.relay.ourdomain.com" address, therefore allowing round robin distribution (not perfect, but still working on a better idea).
6. If the domain doesn't have a "mapped" address in our DNS, it will round robin against a record called "relay.ourdomain.com" which essentially maps to all cluster nodes until the domain is captured and mapped.
7. if none of our servers are able to handle mail for the domain, then we create a "reject" rule in the transport maps so that it sends back an immediate bounce.
8. we track non-existent domains and email address as much as possible, and create rules for them as well as well to prevent continued attempts to send to a bogus address (causing us to be blacklisted)
9. presently due to the volume of email being sent out, and importance of getting out at times tens of thousands of messages to the same provider, we employ 10 clustered servers and when known rate limiting based on provider's own policies.

Ideally if some or all of these protocols could make their way into postal, I'd move over immediately as I do like the idea of things like "webhooks", "inbound processing" and such.

Hope these details help!

*** Original purpose of this thread still remains ***