The available memory of the Kong machine suddenly dropped from 18G to 6.12G

[fairyqb]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

kong all

Current Behavior

The available memory of the Kong machine suddenly dropped from 18G to 6.12G, and then recovered on its own. What are the possible reasons?

During this period, the number of requests did not increase, and other monitoring indicators were normal. Only the available memory of the machine fluctuated greatly.

Expected Behavior

What are the possible reasons?

Steps To Reproduce

No response

Anything else?

No response

[joelact]

Hey @fairyqb!
Do you have metrics about the memory consumed by the kong + nginx workers during that period? Do you have other services deployed on that machine?

[chobits]

It's difficult for us to determine the cause with only the physical memory stats provided.

I think you could also add some metrics to monitor the kong's lua/shdict memory statistics, like using this command:

$ curl -s localhost:8001/status| jq|pbcopy
{
  "database": {
    "reachable": true
  },
  "memory": {
    "workers_lua_vms": [
      {
        "pid": 83261,
        "http_allocated_gc": "47.10 MiB"
      }
    ],
    "lua_shared_dicts": {
      "kong": {
        "capacity": "5.00 MiB",
        "allocated_slabs": "0.05 MiB"
      },
      "kong_locks": {
        "capacity": "8.00 MiB",
        "allocated_slabs": "0.05 MiB"
      },
      "kong_healthchecks": {
        "capacity": "5.00 MiB",
        "allocated_slabs": "0.05 MiB"
      },
      "kong_cluster_events": {
        "capacity": "5.00 MiB",
        "allocated_slabs": "0.05 MiB"
      },
      "kong_rate_limiting_counters": {
        "capacity": "12.00 MiB",
        "allocated_slabs": "0.06 MiB"
      },
      "kong_core_db_cache": {
        "capacity": "128.00 MiB",
        "allocated_slabs": "0.28 MiB"
      },
      "kong_core_db_cache_miss": {
        "capacity": "12.00 MiB",
        "allocated_slabs": "0.06 MiB"
      },
      "kong_db_cache": {
        "capacity": "128.00 MiB",
        "allocated_slabs": "0.23 MiB"
      },
      "kong_db_cache_miss": {
        "capacity": "12.00 MiB",
        "allocated_slabs": "0.06 MiB"
      },
      "kong_secrets": {
        "capacity": "5.00 MiB",
        "allocated_slabs": "0.05 MiB"
      },
      "kong_dns_cache": {
        "capacity": "1.00 MiB",
        "allocated_slabs": "0.06 MiB"
      },
      "prometheus_metrics": {
        "capacity": "5.00 MiB",
        "allocated_slabs": "0.05 MiB"
      }
    }
  },
  "server": {
    "connections_writing": 2,
    "connections_waiting": 0,
    "total_requests": 7,
    "connections_active": 2,
    "connections_handled": 7,
    "connections_reading": 0,
    "connections_accepted": 7
  }
}

[chronolaw]

What is your kong version? What is the deploy mode (traditional or hybrid)? All these information are valuable for trouble shooting.

[github-actions]

This issue is marked as stale because it has been open for 14 days with no activity.

[github-actions]

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks.
We're sorry that your issue could not be resolved. If any new information comes up that could
help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look
our pledge to the community
for more information.

Sincerely,
Your Kong Gateway team