Minimizing memory usage with ORT models using Ort::SessionOptions, RunOptions, Allocator, etc.

[sevagh]

In a memory-constrained environment (e.g. WebAssembly with a max of 4 GB), I would like to reduce the memory usage of my ORT model as much as possible.

Note that I am building onnxruntime for WASM with only simd128, no pthread (I do my own threading with my web worker by splitting the workload among multiple module instantiations).

These are my current settings:

    model.session_options = Ort::SessionOptions();

    // Memory optimizations
    model.session_options.DisableMemPattern();
    model.session_options.DisableCpuMemArena();
    model.session_options.SetExecutionMode(ExecutionMode::ORT_SEQUENTIAL);

    // Thread settings (my WASM build is single-threaded anyway)
    model.session_options.SetIntraOpNumThreads(1);
    model.session_options.SetInterOpNumThreads(1);

    // Arena configuration
    model.session_options.AddConfigEntry("session.use_memory_arena", "0");
    model.session_options.AddConfigEntry("session.use_arena_allocation", "0");

    // General optimizations
    model.session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_DISABLE_ALL);

I also have a global Allocator object at the top of my cpp file which I use for creating tensors, etc.:

// in my h file
namespace global_allocator {
    extern Ort::AllocatorWithDefaultOptions allocator;
}

// in my cpp file
namespace global_allocator {
    Ort::AllocatorWithDefaultOptions allocator;
}

So far from my experiments, this has reached the best/lowest memory usage (e.g. enabling CPU memory arena blows up the memory usage right off the bat).

Are there other options I could be using? How about RunOptions? Is there an allocator.AggressiveCleanup() I should use after calling session.Run()? Etc.

My model only loads a single Ort::Session once, uses it for 1 or many passes of inference.

Any help is appreciated!