New mapped_private mode: avoid crash by flushing dirty pages when memory pressure gets high.

[greg7mdp]

The Linux kernel assigns and maintains an Out of Memory (OOM) score on a per-process basis, The valid range is from 0 (never kill) to 1000 (always kill), the lower the value is, the lower is the probability the process will be killed.

We can track this score for Leap, and when the score gets high, flush the dirty pages of our private mapping to disk, lowering the Leap process OOM score, and thus preventing the system from killing our process.

This would make leap function in a somewhat optimal mode, using as much memory as possible for its chainbase db in order to preventing constant writebacks to disk, while still doing occasional writebacks (to avoid crashing) on systems with limited RAM.

[heifner]

We have a number of caches and will likely have more. Maybe consider a registry of std::function to call when memory is low. One would flush chainbase, one could flush the net_plugin unlinkable_block_state_cache, etc.

[matthewdarwin]

Consider adding prometheus metrics to keep track of how often certain functions are used.

[spoonincode]

We have a number of caches and will likely have more. Maybe consider a registry of std::function to call when memory is low. One would flush chainbase, one could flush the net_plugin unlinkable_block_state_cache, etc.

I am not certain the oom_score is a proper indication of system wide pressure. I would like to see more research before it's used more broadly.

[greg7mdp]

For those interested, I wrote a small program to check oom scores on linux.
For example to start 5 processes, each allocating and accessing 2GB blocks at 1 second intervals, you could use for i in {1..5}; do ./a.out 2 $i & done

#include <cstdint> /* uint64_t  */
#include <cstdlib> /* size_t */
#include <cassert>
#include <optional>
#include <iostream>
#include <fstream>
#include <chrono>
#include <thread>
#include <vector>
#include <memory>

static std::optional<int> read_oom_score() {
   std::ifstream oom_score_file("/proc/self/oom_score");

   if (!oom_score_file.is_open())
      return {};

   int oom_score;
   if (!(oom_score_file >> oom_score))
      return {};

   return oom_score;
}

static void access(std::unique_ptr<char[]>& p, size_t sz) {
   for (size_t i=0; i<sz; i += sysconf(_SC_PAGE_SIZE))
      p[i] = 1;
}

static std::unique_ptr<char[]> allocate_and_access(size_t sz) {
   auto p = std::make_unique<char[]>(sz);
   access(p, sz);
   return p;
}

int main(int argc, char **argv)
{
   using namespace std::chrono_literals;
   size_t block_sz = 1ull << 30; // default 1GB
   size_t num_gb = 1;
   size_t id = 0;
   std::vector<std::unique_ptr<char[]>> v;
   
   if (argc >= 2) {
      num_gb = atol(argv[1]);
      block_sz *= num_gb;
   }
   if (argc >= 3)
      id = atol(argv[2]);

   std::this_thread::sleep_for(500ms);
   for (size_t i=0; i<10; ++i) {
      auto oom = read_oom_score();
      if (oom) {
         std::cout << id << " oom_score after allocating " << i * num_gb << " GB: " << *oom << '\n';
      }

      v.push_back(allocate_and_access(block_sz));
      std::this_thread::sleep_for(1s);
   }
   return 0;
}

[bhazzard]

Needs additional design and research that won't make a Leap 5 timeline.