converting very large obj file to glb

[pramodwerea]

I am trying to convert a 4GB .obj file into GLB but after executing some minutes it gives a message, KILLED. There is no other error message thrown. However, for the smaller size .obj file, it works fine. What could be the possible reason for this?

P.S. I am using glbpack for the ubuntu.

[zeux]

What probably happens is that the system runs out of memory, and OOM killer kills the process. You can probably set up a larger overcommit... How much RAM does your system have?

[pramodwerea]

I have 15GB of memory free out of 15GB, but still, I see the same issue.

Okay, I noticed that executing the conversion command seems it consumed all the memory, thats why process is KILLED. But is this normal to consum that much memory?

[zeux]

While I don't have a 4 GB .obj file to test, a 275 MB .obj file I have on hand gets converted to .glb with peak memory consumption of 924 MB, so it looks like the ratio is ~3.3x, and for your 4GB file to get converted you'll need at least ~13 GB memory. You can probably configure swap/overcommit to get around this limitation.

I'll take a look at whether memory consumption can be improved significantly for .obj path but I suspect it's non-trivial to do so -- the file I'm testing has 13.5M input vertices, and with position/normal data allocated for it (each takes 16 bytes due to how temporary data is represented) this alone requires ~432 MB for vertex data alone, and some processes currently may require double that amount.

My hunch is maybe it's possible to cut the memory consumption by 1.5x but it's unlikely it can be reduced much further, so either way you'll need 10+ GB of RAM to convert a 4 GB file. I'd suggest splitting the .obj file into multiple smaller files before conversion, as it's typical for processing tools to require more memory than the input or output file size alone suggests.

[zeux]

Looking at this further, there's probably some significant gains that could be had depending on the characteristics of the file. It would help if you gather stats for the file so that it's easier for me to optimize the more significant issues, can you run this command and print the output?

cat path-to-file.obj | cut -f 1 -d ' ' | sort | uniq -c

[pramodwerea]

@zeux Thank you for the update.

[zeux]

I recommend trying the build from https://github.com/zeux/meshoptimizer/actions/runs/2728629022 - if it still crashes with the out of memory error, it would be nice to see the output above, as well as the output of gltfpack (before the crash) when launched with -vv argument.