Fixit Week 2023-08: Reduce verbosity of logging in containers #847
Comments
|
After considering the best way forward, I think the first pass for this will be to leverage Vector filtering rules to eliminate a bunch of common info/debug log message from most services that will account for the majority of the log messages. This is easy and doesn't touching any code. Additionally, in the short to mid term, this also leaves all the current logging available in container logs and k8s, so that they can be leveraged for debugging. It will only affect which logs get pushed to Cloud Logging. The longer term solution will probably be to use a leveled/structured logger for all of M-Lab's services so that they can be properly filtered and displayed in various environments. |
This is a first stab at trying to filter log messages from 3rd-party experiments, as well as overly verbose messages from sidecars. #847
…#848) * Filters unwanted logs messages from GCP This is a first stab at trying to filter log messages from 3rd-party experiments, as well as overly verbose messages from sidecars. #847 * YAMLize multiple conditions I'm not sure if this is the right syntax. Documentation on using multiple conditions in YAML is sparse to non-existent. * Renames Vector filter transforms Vector threw this error: Configuration error. error=More than one component with name "kubernetes_logs" (source, transform) * Updates sinks input names to match transform names * Moves dot inside double quotes in filter transform * Ands expression, and removes quotes from left sides * Tries a different syntax for filter using match() * Casts .message as a string Got this from Vector: .message: this expression resolves to any, but the parameter "value" expects the exact type string # Please enter the commit message for your changes. Lines starting * Adds 'abort' expression to fallible function * Casts 'container_name' to string
|
PR #849 closes this issue. We ultimately decided to not push any experiment container logs to GCP, by default. If we need to in the future, then it should be as easy as changing the pod label |
The spend on log storage in GCP is non-trivial. The logs are polluted with mostly useless debug information, making them hard to look through for log entries of interest. The software we use to upload container logs (Vector) is exceedingly busy, possibly triggering latent, edge-case bugs.
The text was updated successfully, but these errors were encountered: