About receiving XML with large-size attached files

[nishiyo]

Dear @phax
May I ask a question about the handling of xml with large-size attached files (eg:png, pdf).

When we receiving an XML with a large attached file, it takes time to process.

• 40MB, with attached file, takes 4min

On the other hand, when we receiving an large XML without any attached file, there are no issues.

• 40MB, without attached files, takes less than 2sec

It seems that the process of signing verification and decryption is taking time...
Are there any differences in the process between cases with and without an attached file?

Also, are there any ways to mitigate this issue?

We are not actually receiving data as described above.
According to Peppol's rules, we are required to capability to send/receive files up to 100MB,
so we are conducting tests using various data.

As additional info,
40MB with attached file, compress with gzip→ 25MB
40MB without attached file, compress with gzip→ 50KB

Best Regards

[phax]

The order of application of the different transformations is:

1. Compress (if needed)
2. Sign
3. Encrypt

So it seems like the test file you used was very good compressible (e.g. by repeating a single character a lot of times), which shrunk down the whole transmission size.

I have made similar tests locally, with up to 200MB and the WS Security stuff is not the problem. The main issue I faced was the network traffic. Because each message typically passes through load balancers, firewalls, reverse proxies, WAFs and the application server until it reaches your application. Usually this is what takes up the time.

The differences between messages with and without attachment is simple. If there is no attachment, the main XML is included into the SOAP Body of the payload, and an "embedded signature" must be calculated.

To really test the decryption/encryption time I suggest you build a scenario that is sending from localhost to localhost, so that network traffic can be excluded. Then you can also easily play around with compression on/off etc.

hth, Philip

[nishiyo]

Thank you for the support!
I'll test in more detail.

[chamikaCN]

Hi @phax,

We encountered something related to this discussion recently. In our production endpoint, we noticed the Long running request message in the logs when we were receiving documents. We also had some of our senders complaining their AS4 clients got timed out (after 2 minutes) trying to send documents to our endpoint.

So we did an investigation and found out there is a correlation between the attachment size and this issue. To irradicate the network factor we tested sending AS4 messages with various attachment sizes in a locally hosted client and server setup. It took just 3 seconds for an as4 message with a single XML attachment to deliver, while for an as4 message with 30MB pdf attachments, it took 2 min 30 seconds causing the client to timeout. We tried to figure out from where the extra time consumption happened when debugging and it was the following method trace.

 boolean signatureOk = xmlSignature.validate(context); SignatureProcessor.java:381
 
 XMLSignature xmlSignature =
            verifyXMLSignature(elem, certs, publicKey, secretKey, signatureMethod, data, data.getWsDocInfo()); SignatureProcessor.java:229
            
List<WSSecurityEngineResult> results = p.handleToken((Element) node, requestData); WSSecurityEngine.java:340

wsResult = processSecurityHeader(elem, requestData);  WSSecurityEngine.java:251

final WSHandlerResult aHdlRes = aSecurityEngine.processSecurityHeader(aSOAPDoc, aRequestData); SOAPHeaderElementProcessorWSS4J.java:179

eSuccess = _verifyAndDecrypt(aSOAPDoc,
                        aAttachments,
                        aState,
                        aProcessingErrorMessagesTarget,
                        WSSConfigManager.getInstance()::createWSSConfig); SOAPHeaderElementProcessorWSS4J.java:503
                        
aProcessor.processHeaderElement (aSoapDocument,
                                             aHeader.getNode (),
                                             aIncomingAttachments,
                                             aState,
                                             aProcessingErrorMessagesTarget) AS4IncomingHandler.java:467
     .......

we tried using a local machine with lesser RAM and it took more than 2 min and 30 seconds for the same document to resolve. Can the Signature verification process be the culprit as the trace shows? if so are there any suggestions you could give us to mitigate this issue?

[sboklund]

Hi @phax, how do we proceed from here? Do you have any ideas? Thanks.

[phax]

Well, my next thoughts are: measure the time on the sending side required for encryption etc. to see if the same pattern applies. In general @sboklund we may consider creating a Service Desk ticket to OpenPeppol to discuss their current SLAs, and that this seems virtually impossible (leaving aside receiver validation). Do you have any ideas that you want to share here?

[chamikaCN]

Hi @phax,

We also did some local testing regarding this issue. and these were the results. It seems to verify the culprit as well.

[chamikaCN]

We managed to track a little bit more using the JProfiler and it revealed that Arrays.fill() usage inside the AESCipher was the root cause. I'm just curious about the approaches we could take to resolve this. Is there a possibility of using a different implementation of WSS4J implementation apart from this Apache library?

[chamikaCN]

Hi @phax,

Continuing on this we were able to unravel some other information related to this issue.

The delay happens only in the process of calculating the digest value of the 3rd reference (the attachment). In that when trying to apply the relevant transform (Attachment-Content-Signature-Transform) CipherInputStream class consumes around 80% of the total process time to prepare the stream before reading it into the buffer.

It appeared to us any efficiency improvement there is much harder to achieve, assuming the time spent is increasing with the size of the encrypted attachment. However, would there be any other efficient alternative to achieve the same result?

Also please let us know if anything else we could do from outside to resolve this issue and what the next steps would be.

Is this a common issue with attachment sizes that are approximately larger than 20MB, or have other known users faced this?

Thanks,

[chamikaCN]

Hi,

We managed to resolve this issue by upgrading the JRE in our deployed instances to JAVA21. With that change, all the previous signature verifications that took 2-15 minutes are done in a matter of a few seconds. Apparently, the underlying relevant Java security library implementations for signature verification have been optimized with this upgrade. We'll post here if we manage to find anything related to this issue and the solution.

[phax]

Cool - thanks. This is very good information :)

[phax]

I can confirm that the reception time with Java 21 is heavily reduced. The below snapshot shows the difference between a Java 11 and a Java 21 run:

Note: in the image above "Old" means "Java 21" and "New" means "Java 11"