Optimize memory usage in the reduce script

[crespocarlos]

Summary

POC to improve the memory consumption of the scripted metrics agg.

Current version

Running the script with the oom.ts scenario caused an Out of Memory (OOM) error. The heap usage spiked to around 96% before crashing.

The problem is caused by the paths object, which is a matrix containing the entire path for all events:

Rows: 8517
Columns (avg): 252 (could be many more)
Fields in each item: 3 (service.name, service.environment, agent.name)
Size of a field in bytes (avg): 55

paths object consumption per ES request (ROUGH ESTIMATION): 8517 rows * 252 columns * (3 fields * 55 bytes/field) = ~355 MB

This amount of data can also lead to a "content length bigger than the maximum allowed string" exception.

Refactored version

With the refactored version, the heap usage appears linear and didn't exceed 90% in any of the 3 calls made.

The query was streamlined to respond with what is necessary for the frontend to render the map in the connections object. This creates a much smaller object that the query has to hold in memory compared to what paths require.

eg:

"service-446~>service-447": {
  "destination": {
      "span.subtype": "unknown",
      "span.destination.service.resource": "service-447",
      "span.type": "app"
    },
    "source": {
      "service.environment": "Synthtrace: service_map",
      "service.name": "service-446",
      "agent.name": "nodejs"
    }
  },

With the oom scenario,connections holds approximately 550 unique connections. Therefore:

Keys: ~550 * 2 (source and destination) = 1100
Fields in each key: 3 (service.name, service.environment, agent.name fields)
Size of a field in bytes (avg): 55

connections object consumption per ES request (ROUGH ESTIMATION): 1100 keys * (3 fields * 55 bytes/field) = ~177KB

Worst-case scenarios, such as a long cyclic map, might still cause the space complexity to be O(N^2) and could potentially lead to an OOM.