-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes pod labels are sometimes missing #1775
Comments
Hi @Namnamseo, there is a new component released with the latest falco version called k8s-metacollector which has been developed for such use-cases. It reduces the cases where the pod metadata is missing. Here you can find the docs on how to install it using the falco chart: https://github.com/falcosecurity/charts/tree/master/charts/falco#k8s-metacollector |
Right, I've seen those. A standalone metadata collector would really bring up the overall stability. I only need the pod labels, so I was wondering if this can be done with just the container runtime integration. |
@Namnamseo once Falco 0.38.0 is out very soon it would be interesting to see if the container runtime socket info extraction is working better since we improved it a bit. And as @alacuku stated you also have the option to use the new k8s plugin. |
Issues go stale after 90d of inactivity. Mark the issue as fresh with Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with Provide feedback via https://github.com/falcosecurity/community. /lifecycle stale |
Stale issues rot after 30d of inactivity. Mark the issue as fresh with Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with Provide feedback via https://github.com/falcosecurity/community. /lifecycle rotten |
Describe the bug
Hi!
I'm using Falco to monitor some specific syscalls of Kubernetes pods on a GKE cluster.
It seemed to work well at first, but I've noticed that some events had incomplete fields.
These events:
k8s.pod.labels
(shows up as<NA>
)k8s.pod.label[some.valid/label]
(shows up as<NA>
)k8s.ns.name
k8s.pod.cni.json
k8s.pod.name
container.id
container.name
container.image.repository
container.image.tag
Upon inspecting the logs, I think that a pod sandbox query is sometimes failing, and it keeps staying that way.
One thing I noticed is, when I restart the Falco pod on that node, it parses the labels fine.
My weak guesses (after quickly skimming through what I've seen) are:
One subtle issue: in these two log lines,
the latter one should explain why PodSandboxStatus (not ContainerStatus) call failed, but there was a bug (using
status
defined earlier instead ofstatus_pod
) in the library v0.14.3.This seems to have been fixed in rigorous refactoring since.
How to reproduce it
Sorry, I couldn't have this consistently reproduced. It occurs from time to time with no pattern.
Expected behaviour
k8s.pod.labels
andk8s.pod.label[some.valid/label]
are always filled.Also, the log should show up like this. (This is the log when everything is normal, and above fields are filled).
Screenshots
Environment
v1.27.11-gke.1202000
$ ctr version
says1.7.12-0ubuntu0~22.04.1~gke1
Ubuntu 22.04.3 LTS
5.15.0-1049-gke
falcosecurity/falco
Helm chart (of version 4.2.3)--disable-cri-async
Additional context
These are some logs I found relevant.
The text was updated successfully, but these errors were encountered: