Monitoring, tracing, observability in DevOps

[monperrus]

• https://en.wikipedia.org/wiki/Tracing_(software)
• https://en.wikipedia.org/wiki/System_monitoring
• https://en.wikipedia.org/wiki/Network_monitoring
• https://en.wikipedia.org/wiki/Crash_reporter
• https://en.wikipedia.org/wiki/Application_performance_management
• https://en.wikipedia.org/wiki/Website_monitoring
• https://en.wikipedia.org/wiki/Provenance#Computer_science
• https://en.wikipedia.org/wiki/Log_analysis

[monperrus]

See also icinga (thanks to @henriklb for the suggestion)

[monperrus]

Log analysis @eclipse https://projects.eclipse.org/projects/tools.tracecompass

[MatsJonsson]

We've found Istio ( https://istio.io/ ) to be increasingly useful in this context. KubeSpy ( https://github.com/pulumi/kubespy )is an excellent tool for troubleshooting and diagnosing Kubernetes deployments.

[lsc]

• Prometheus
• Sensu https://sensu.io/
• Zipkin
• the ELK stack.

[MatsJonsson]

+1 for Prometheus

[bittermandel]

Sentry for Error Reporting. https://sentry.io/welcome/

[monperrus]

• OpenZipkin
• Jaeger https://github.com/jaegertracing/jaeger
• https://medium.com/@rakyll/cpdd-critical-path-driven-development-6c2592fb8ea4

(from #16 (comment))

[monperrus]

See also Runtime application self-protection #18 (comment)

[monperrus]

Analytics

[monperrus]

Tools and Benchmarks for Automated Log Parsing.
http://arxiv.org/abs/1811.03509

[monperrus]

Does the Fault Reside in a Stack Trace? Assisting Crash Localization by Predicting Crashing Fault Residence
https://www.sciencedirect.com/science/article/pii/S0164121218302401

[monperrus]

Having good dashboards is essential in DevOps, see Kibana, etc.

[monperrus]

Made in Alibaba: https://github.com/alibaba/Sentinel

[monperrus]

JVM Profiler Sending Metrics to Kafka (https://kafka.apache.org/), Console Output or Custom Reporter
https://github.com/uber-common/jvm-profiler

[monperrus]

https://github.com/madflojo/automatron

[monperrus]

https://github.com/apache/incubator-skywalking

[monperrus]

Time-series database to store monitoring data
https://en.wikipedia.org/wiki/Time_series_database

[monperrus]

Prometheus - Monitoring system & time series database
https://prometheus.io/

[monperrus]

Netflix Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more.
https://github.com/Netflix/zuul

[monperrus]

OpenTracing
https://opentracing.io/

[monperrus]

Nagios
https://en.wikipedia.org/wiki/Nagios

[monperrus]

Sensu is a free and open source monitoring that handles cloud environments. Sensu allows you to monitor servers, services, application health, and business KPIs.
https://xebialabs.com/technology/sensu/

[bbaudry]

Provenance analysis tools

• SPADE : https://github.com/ashish-gehani/spade
• Camflow : http://camflow.org/

[monperrus]

Framework for instruction-level tracing and analysis of program executions
http://static.usenix.org/event/vee06/full_papers/p154-bhansali.pdf

[monperrus]

DevOps Metrics
https://queue.acm.org/detail.cfm?id=3182626

[monperrus]

Dapper, a large-scale distributed systems tracing infrastructure at Google
http://research.google.com/pubs/pub36356.html

[monperrus]

Contemporary Software Monitoring: A Systematic Mapping Study.
http://arxiv.org/pdf/1912.05878

[monperrus]

Cilium - eBPF-based Networking, Observability, and Security
Cilium's control plane is highly optimized, running in Kubernetes clusters of up to 5K nodes and 100K pod
https://cilium.io/

[monperrus]

Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. Can be used for monitoring events. Can be bridged with MQTT.
https://aws.amazon.com/kinesis/data-streams/

[monperrus]

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.

Can be used to feed Prometheus.

https://micrometer.io/

[gluckzhang]

Prometheus client libraries (including both official ones and many third-party ones) can be found here: https://prometheus.io/docs/instrumenting/clientlibs/

[monperrus]

Paper: "Enjoy your observability: an industrial survey of microservice tracing and analysis" http://link.springer.com/10.1007/s10664-021-10063-9

[monperrus]

Faaster troubleshooting-evaluating distributed tracing approaches for serverless applications

[monperrus]

Timeloops: System Call Policy Learning for Containerized Microservices.

[bbaudry]

Sampler is a tool for shell commands execution, visualization and alerting.
Configured with a simple YAML file.
https://sampler.dev/

[monperrus]

Stagemonitor is a Java monitoring agent that tightly integrates with time series databases like Elasticsearch, Graphite and InfluxDB to analyze graphed metrics and Kibana to analyze requests and call stacks

https://github.com/stagemonitor/stagemonitor

cc/ @gluckzhang

[monperrus]

Trace Server Protocol
https://github.com/eclipse-cdt-cloud/trace-server-protocol

[bbaudry]

a sweet feature of grafana
https://grafana.com/blog/2021/07/30/how-to-use-grafana-and-prometheus-to-rickroll-your-friends-or-enemies/?src=li&mdm=social

[bbaudry]

https://github.com/MacroPower/prometheus_video_renderer

[bbaudry]

Zabbix open source monitoring solution for network monitoring and application monitoring of millions of metrics.
https://www.zabbix.com/

[monperrus]

strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.
https://strace.io/

[monperrus]

Spring Metrics
https://docs.spring.io/spring-metrics/docs/current/public/prometheus

[monperrus]

Let's Trace It: Fine-Grained Serverless Benchmarking using Synchronous and Asynchronous Orchestrated Applications
https://arxiv.org/pdf/2205.07696.pdf

[monperrus]

Open Tracing Tools: Overview and Critical Comparison
https://arxiv.org/pdf/2207.06875.pdf

[monperrus]

Reliability Pillar - AWS Well-Architected Framework
https://docs.aws.amazon.com/pdfs/wellarchitected/latest/reliability-pillar/wellarchitected-reliability-pillar.pdf

[monperrus]

Towards Solving the Challenge of Minimal Overhead Monitoring. (arXiv:2304.05688v1 [cs.SE])

[monperrus]

Lessons Learned Building a Global Synthetic Monitoring System
Talk at SREcon
https://www.usenix.org/conference/srecon22apac/presentation/sidh

[monperrus]

Elastic Observability
https://www.elastic.co/observability

[monperrus]

Dynatrace
https://www.dynatrace.com/solutions/application-monitoring/

[monperrus]

CausalRCA: Causal inference based precise fine-grained root cause localization for microservice applications (JSS 2023)

[bbaudry]

A curated collection of publicly available resources on how technology and tech-savvy organizations around the world practice Site Reliability Engineering (SRE)
https://github.com/upgundecha/howtheysre

[bbaudry]

Observability with Gitlab
https://opstrace.com/
https://about.gitlab.com/direction/monitor/observability/