Downloading folders with many small files stuck for very long time in "Prepare <filename> (Overwrite)" followed by very slow transfer speed or dropped connection

[lostminds]

This ticket system is to report bugs and feature requests. For support, visit the help page first.

Describe the bug
A clear and concise description of what the bug is.

A long standing issue I've had with Cyberduck is that it's very slow when downloading folders over SFTP with many files (typically things like website plugin libraries etc, try the https://matomo.org/ library for example, it's about 180MB but contains over 12000 files). When trying to download such a directory Cyberduck seems to get stuck in some preparation stage where it's listing the files to download for a very long time (10min+) "Prepare (Overwrite)" even though there's no overwriting and it's just downloading to a new directory.

When this process finishes there seems to be some sort of overhead per file download start, resulting in very long download times. And during this process (or sometimes even before during the long listing phase) it will now lose the connection to the server.

Downloading a single 100MB file from the same server will take seconds, while downloading 10000 small files and folders of the same size will take 30+min and often take multiple attempts due to connection failures. With earlier versions of Cyberduck I was usually able to get the download after a couple of tries, but with the current version it fails every time if the number of files is to great. Often it seems to get stuck in a state where it says "Retry again in X seconds (1 more attempt)" where it will reconnect, "Prepare" one more file, fail and retry again over and over and over without ever getting to actually start downloading anything.

When you then stop it to try again it gives you the following error message (which I think may also be incorrect):
"Listing directory upgradephp failed.
Connection or outbound has closed. The connection attempt was rejected. The server may be down, or your network may not be properly configured."
Since the connection was stopped by me and my network connection is still active.

To Reproduce
Steps to reproduce the behavior:

1. Get a folder with many small files and subfolders (for example the https://matomo.org/ library) and upload this to your server.
2. Attempt to download folder again via SFTP and observe very long processing times and dropped connections.
3. If/when it gets stuck, try stopping the transfer and get confusing / incorrect error message.

Expected behavior
Folder downloads with all files.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: macOS 15.0.1
• Version 9.0.2 (42108)

Log Files
Please include the log file. You can reach the logging output in Preferences → Connection.

I'm guessing there were a lot of log entries for all these files (and possibly issues) so the log was 37MB and over the limit of what I could upload here. So I trimmed it down to the last 10000 or so lines, that should hopefully be enough.

cyberduck_log_trimmed.log

Additional context
The performance issues of downloading folders with many files is not new for me, but the problem has gotten worse and it's now failing and dropping the connection instead of just being slow.

[lostminds]

Additionally if the process fails during such a long slow download with an error about directory listing failed you have an option to try again. If you do, the transaction seems to end up in some invalid state, starting first to actually download files, but with no progress bar and seemingly missing the transaction context of the total size as it now says "out of 0B".

After a while, it will revert again to "Prepare" but this time "Prepare (Resume)" going through a lot of directories before starting to download again. But still not getting the total size of the operation it's doing right:

And eventually when it finishes it says the transaction was incomplete 8.3 of 7.3 MB (even if it is now finished and has downloaded 77.3 MB total)