deis-nsqd pod is leaking FS usage

[Cryptophobia]

From @felipejfc on March 19, 2017 2:22

posted in

deis/logger#138

Copied from original issue: deis/nsq#15

[Cryptophobia]

From @felipejfc on March 29, 2017 17:55

Think this happens when kubernetes has too many logs/s, due to nsq's architecture, I believe it will keep writing to disk if the logger does not consume from it fast enough. This leads to the leak.
Maybe we should scale the nsq and logger?

[pdomagala]

@Cryptophobia Any update on it? I came across on this issue last night. My deis-nsqd pod used 500GB of storage. Should I scale it up?

[Cryptophobia]

@pdomagala , can you give more details on the issue? I suspect your applications are spewing lots of logs and nsqd cannot keep up. You can either limit their log level or scale up. Have you seen how much memory and CPU nsqd needs, it may also need more CPU to handle the load.

[pdomagala]

@Cryptophobia To be honest, we migrated our k8s nodes to faster storage (NVME) and scaled up nsqd replicas and problem has gone.

[Cryptophobia]

Great to hear! Must have been a problem with scale or the storage IO speed.