[EKS] [Managed Workers]: Send kubelet logs to CloudWatch

[aaron-trout]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Would be great for kubelet / other managed worker node logs to be sent to CloudWatch.

Which service(s) is this request for?
EKS (Managed worker nodes)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Developers are already using the control plane logs (specifically the audit log) to assist in debugging, but occasionally the platform team has to step in and SSH into worker nodes to pull kubelet logs.

It would be super helpful if these were sent to CloudWatch like the control plane logs.

Are you currently working around this issue?
Workaround is to just SSH to the worker node, but obviously this has some limitations, for example when using cluster-autoscaler the node might not live for very long.

[tpsk-hub]

Hi Folks,

We now have the ability to solve the above ask. Check out this blog to learn more - https://aws.amazon.com/blogs/containers/fluent-bit-integration-in-cloudwatch-container-insights-for-eks/

[roberto-civitas]

I dont see how that addresses the above

[deepankarm]

Would be great for kubelet / other managed worker node logs to be sent to CloudWatch.

Is this already doable?

[joeynaor]

When had an EKS node unexpectedly changing its status to notReady. As far as I understand, the only way to find out the reason behind (beside running kubectl describe <node> at the time of failure) this is by checking kubelet logs (journalctl -u kubelet), which in our case rotated out. Being able to log these conveniently is a must for RCA purposes.

[michaelswierszcz]

same situation happened with us recently

When had an EKS node unexpectedly changing its status to notReady. As far as I understand, the only way to find out the reason behind (beside running kubectl describe <node> at the time of failure) this is by checking kubelet logs (journalctl -u kubelet), which in our case rotated out. Being able to log these conveniently is a must for RCA purposes.

[tooptoop4]

did u solve @joeynaor @michaelswierszcz ? i'm thinking related to aws/amazon-vpc-cni-k8s#2808

[michaelswierszcz]

ended up scraping kubelet logs with our exists logging infrastructure (fluent-bit -> loki)

[joeynaor]

@tooptoop4 Our workaround was to disable "delete on terminate" for the EKS nodes disks. After an incident, we mounted the disk of the faulty node to a regular EC2 instance and inspected the logs. In our case, the only incident ever since was caused by a hardware failure on AWS side, confirmed by AWS support.