This repository has been archived by the owner on Apr 24, 2023. It is now read-only.
Kustomize base for fluent-bit #90
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Remove a duplicate line.
Signed-off-by: Julien Vincent <[email protected]>
…tes-logging into fix/nginx-parser-regex
Signed-off-by: Harrison Heck <[email protected]>
Signed-off-by: Karl Skewes <[email protected]>
Signed-off-by: David J. M. Karlsen <[email protected]>
It's safe to add apps/v1 to Daemonset as it's stable from 1.9, Which I think almost all are. Signed-off-by: rjshrjndrn <[email protected]>
…to reference Kafka version Signed-off-by: Igor Yankovsky <[email protected]>
This reverts commit a6ec71d.
kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
Would be interesting to keep fluent-bit log forwarding for the duration of shutdown. https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/ https://kubernetes.io/docs/concepts/architecture/nodes/#graceful-node-shutdown https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms
Invdividually overridable files, using "behavior: merge" Note that with "disableNameSuffixHash: false" (the default) we can roll back configuration changes.
and prefixed by the tag we document what we pulled
solsson
added a commit
to Yolean/unhelm
that referenced
this pull request
Aug 10, 2021
Displaying unparsed log entries using kubectl apply -k base; kubectl rollout status daemonset fluent-bit; kubectl logs -f -l app.kubernetes.io/name=fluent-bit Issues like fluent#88 and fluent/fluent-bit#1902 (comment) indicate that depending on /var/log/containers symlinks cause quite a few issues. /var/log/pods/ is the path stated in https://github.com/kubernetes/kubernetes/blob/v1.22.0/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L63 and I've verified on GKE cos-containerd, GKE ubuntu-dockerd and k3s that the path contains the actual files, not symliks. Also using /var/log/pods makes it trivial to exclude logs from any container named fluent-bit. Doing so reduces the risk of endless log loops.
I find no benefits with the stricter regexes used in most examples, as long as paths are well defined.
Looks like the default format of the file output plugin
|
I find the |
which helps human consumption when log records are json
from tcp_transport.go:313 Use PodMonitor etc. instead.
examples but I think they are two extremes where the former requires custom parsing to be fully useful and the latter makes no case for the Fluentbit data pipeline at all without a consuming step.
About the "toil" image, grafana/logcli doesn't have curl and I wanted to provide an example of how to consume logs from Loki. We echo out the logcli commands instead and do everything with curlimages/curl
solsson
changed the title
|
Work is complete, but obviously needs validation by others. A merge to master would affect online docs which I suppose is quite an undertaking. It's been interesting to see how far across blogs and helm charts my early yaml from this repo has proliferated. For example when researching I've seen the unconventional volume name I think this PR meets its three goals (above). I'll keep it as default branch in https://github.com/Yolean/fluent-bit-kubernetes-logging for easy access to example yaml, regardless if some of this work is merged or not. We're upgrading our production setup, which had degraded quite a bit, to use the online base. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've gone through the PRs filed since I participated in early yaml work here, and unsurprisingly the duplication is a maintenance problem. I think Kustomize is a good fit for eleminating that duplication.
Log processing is prone to heavy customization. I see a few use cases for this repo:
A Kustomize base that in itself isn't functional is a good start for all three of them. Operators and stack-specific helm charts like Grafana's may be more widely used for production, but if we can avoid duplication in this repo it IMO offers a straightforward way for the community to share experiences.