Very slow export (Please enable parallel compression)

[albert-a]

Hello,

GNS3 server is installed in Debian 12 LXC container in Proxmox 8. Container resources: 72 cores / 128g RAM
GNS3 version: 2.2.43
I have a project with 10 installed Proxmox 8 VMs. Project uncompressed size on the GNS3 server is 70G.

I'm experiencing extremally low export speed:

Export to ZIP file (without base images / snapshots) : size=30G duration=57m
Export to LZMA file (without base images / snapshots) : size=17G duration=3h27m

Steps to reproduce the behavior:

On client go to Menu -> File -> Export portable project -> choose path, compression algorithm -> Next -> Finish

My investigation

Network speed is not an issue, copy speed from server to client: 120Mb/s
I decided to measure compression speed from the command line (I compressed the same project):

single thread ZIP compression

zip -r out.zip PROJDIR - size=30G duration=43m

single thread LZMA compression

tar cJf out.tar.xz PROJDIR - size=18G (18297M) duration=3h7m

multithreated LZMA compression

tar c PROJDIR | xz -T0 > out.tar.xz - size=19G (18787M) duration=9m10s
tar c PROJDIR | xz -T0 -9 > out.tar.xz - size=17G (17487M) duration=9m23s - I THINK IT'S THE BEST FOR LZMA

multithreated ZSTD compression

tar c PROJDIR | zstd -T0 > out.tar.zst - size=23G (22983M) duration=1m44s
tar c PROJDIR | zstd -T0 -6 > out.tar.zst - size=22G (22297M) duration=1m46s
tar c PROJDIR | zstd -T0 -9 > out.tar.zst - size=21G (20745M) duration=2m2s - I THINK IT'S THE BEST FOR ZSTD
tar c PROJDIR | zstd -T0 -12 > out.tar.zst - size=21G (20697M) duration=3m40s
tar c PROJDIR | zstd -T0 -15 > out.tar.zst - size=21G (20473M) duration=6m28s
tar c PROJDIR | zstd -T0 -19 > out.tar.zst - size=19G (18834M) duration=11m
tar c PROJDIR | zstd -T0 --zstd=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6 > out.tar.zst
- size=20G (19560M) duration=7m49s

Conclusion

I suppose the problem is in the compression. Jujing by the speed, currently export implemented with the single threaded compression.

I think parallel ZSTD with compression level 9 zstd -T0 -9 is the best option.
Parallel LZMA with compression level 9 xz -T0 -9 must also be kept for maximum ratio

I'm asking to enable parallel compression during export.
It could save hours of waiting.

Best regards,
Albert

[grossmj]

We have added ZSTD compression in version 3.0 (currently in beta). I will have to check if we have parallel compression.

[albert-a]

Thanks you! This would really be a timesaver - x72 speed improvement for my use case.