Expected throughput of phase4 peppol AP

[Jef-VDD]

Hello @phax,

We have recently started using the phase4 library to send documents over the peppol network and were wondering if you have any benchmarks for expected throughput of documents to the peppol network.

Currently, we're using a single EC2 instance with the following reserved resources:
CPU: 0.5 vCPU
Memory: 2 GB
The documents we are testing with are invoice ubl's of about 20-30kb.
Our observed throughput is around 45-60 documents per minute. We're reaching out to see if this aligns with any benchmarks or if there are potential optimizations we might be overlooking.

Any insights you could share would be greatly appreciated.

Thanks for all the work in the peppol ecosystem!

[phax]

@Jef-VDD I don't have ready made benchmarks, because the systems are so different and the numbers are hard to compare.
Most of the time is usually lost in networking, so the bigger the messages, the smaller the throughput.

May I ask for what scenario you are planning to use it? Does the scenario differentiate between compression, encryption and signing? Or is it all mandatory?

I have a test scenario where I am sending 500 messages over the Internet in 10 parallel tasks. That today took a total of 5 minutes and 14 seconds (so ~0,63 secs per dcument). If you are interested, I could run the same test with a receiver on localhost to remove the Internet overhead....

[Jef-VDD]

Hello @phax

I have done some more testing and given some more resources to my access point. Currently :
CPU: 2 vCPU
Memory: 4 GB
Test scenario is the same invoice UBL with validation enabled on my access point. The receiving is done by the same access point instance.

When running a single thread we see it takes 0.5s per document (throughput of 2/sec).
We can keep this ~0.5s per document up until 3 to 4 concurrent threads.

But if we then add any number of threads after 4 we seem to be hitting a throughput limit of 5 documents per second. Average time per document increases linearly.

I assume after about 4 threads we hit 100% cpu and adding more threads wil offer no more benefits on throughput.

Does this align with your expectations or should we be able to use a higher number of threads with these resources?

We are trying to get a rough impression on the resources we need per document to scale our throughput properly.

[phax]

As stated above: I don't have clear expectations from my side, as the list of potential parameters is too high. In general I am a big fan of clear answers, but in this particular case I can't give you one

• Try the same test with a 2 MB document and you will see different results. Try even 100 MB and it becomes worse.
• It depends on what "vCPU" you are running
• It depends on what Java version you are running
• It depends on what you do with the inbound message
• It depends on the configuration of your Loadbalancer, etc. on the number of parallel requests, throughput etc.
• It depends on your web application firewall

In general, I would say 4GB is too little for a large scale setup. Give it at least 16GB so that you are able to handle 2 concurrent 100MB requests. If that can be achieved within a 60 second time limit you are good.

hth

[Jef-VDD]

Thanks again for your insights and the quick response! We understand the complexity of our questions and recognize the multitude of factors involved does not result in a clear answer. We're struggling with the numerous variables ourselves so any input is greatly appreciated, I will give run some tests again with 16GB.

[phax]

Quick hint: others run the solution in AWS as follows:

• Behind the API Gateway or LB, use a Fargate service
• That Fargate Service writes the incoming request "as is" (!!) to S3
• Fargate triggers a Lambda to process the inbound message, by passing on the path to S3
• The response from the Lambda is synchronosuly returned via Fargate to the caller

This solution deals with the 100MB limit, is relatively cheap, and includes an automatic "dumping" of the request. Additionally the Fargate service can be made scalable in a way

(Please excuse me, if I am not 100% accurate - my AWS knowledge is limited)