Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/fluentd #59

Merged
merged 38 commits into from
Aug 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
4175273
Initial commit forwarder/aggregator architecture
ricsanfre Jun 24, 2022
dc42dfa
Changing Fluentd-aggregator imagePullPolicy and keeping fluentd-daemo…
ricsanfre Jun 25, 2022
3bfd3b0
Adding custom fluentd-aggregator image and refactoring ansible tasks …
ricsanfre Jun 28, 2022
0a7beb0
Configuring fluent-elasticsearh-plugin with log_es_400_reason set to …
ricsanfre Jun 28, 2022
46e5030
Remove ansible variables and jinja2 code to create fluentbit configur…
ricsanfre Jun 29, 2022
b7eff6f
Indentation issue solved in prometheus-stack chart config
ricsanfre Jun 29, 2022
f8d308a
Updating logging documentation to include forward/aggregator architec…
ricsanfre Jun 30, 2022
13736a5
Securing communications between fluentbit forwarders and fluentd aggr…
ricsanfre Jul 2, 2022
71b363e
Exposing fluentd forwarding port as cluster service and configuring f…
ricsanfre Jul 2, 2022
099b60a
Updating fluentd fluentbit configuration
ricsanfre Jul 3, 2022
44f35d3
Restructuring log management documentation. Split in several pages
ricsanfre Jul 3, 2022
db0bd68
Changing fluent agent/sidecar pattern name to forwarder-only
ricsanfre Jul 17, 2022
f365afd
Update logging documentation: adding different log forwarding archite…
ricsanfre Jul 20, 2022
965f27b
Fix #54. Updating Metallb installation procedure to make it work with…
ricsanfre Jul 21, 2022
255c509
Removing fluentd forwarding rule. It is not needed. Updating firewall…
ricsanfre Jul 21, 2022
c396539
Updating fluentbit role to release 1.0.4
ricsanfre Jul 21, 2022
e2eb881
Fixing typo in SAN installation document
ricsanfre Jul 22, 2022
df77851
Logging solution documentation update
ricsanfre Jul 22, 2022
3279e25
Fixing typo in hardware documentation
ricsanfre Jul 22, 2022
99240fc
Fix #55. Removing K3s controller-manager and scheduler deprecated args
ricsanfre Jul 23, 2022
9e3f090
Fluentbit: enabling filesystem buffering
ricsanfre Jul 23, 2022
a39d4d5
Prometheus: adding fluent-bit monitoring
ricsanfre Jul 23, 2022
fcfb92a
Adding monitoring of fluentbit storage metrics
ricsanfre Jul 24, 2022
1688315
Prometheus adding fluentd metrics
ricsanfre Jul 25, 2022
df9eb1d
Prometheus elasticsearch exporter installation and prometheus monitor…
ricsanfre Jul 26, 2022
320da09
Updating logging documentation
ricsanfre Jul 27, 2022
4c230da
Closes #53: Configuring Kubernetes' graceful shutdown feature
ricsanfre Jul 27, 2022
8f729af
Fix #58: Solving issue with confliting fields in ES
ricsanfre Jul 28, 2022
52c5d53
Fluentd: Update installation using official helm chart
ricsanfre Jul 30, 2022
1f49b4a
Logging doc update: Adding forwarder only architecture installation i…
ricsanfre Jul 31, 2022
53428a2
Adding proper labels to kubectl wait for pod commands
ricsanfre Aug 1, 2022
7e7be61
Fix #57. Adding support for processing multiline/stack trace log mess…
ricsanfre Aug 1, 2022
29fc97f
Configuring gateway fluentbit node exporter and prometheus to collect…
ricsanfre Aug 2, 2022
e9f7cd6
Doc index. Updating software versions tested
ricsanfre Aug 2, 2022
b4b8452
Web: changing post pages layout
ricsanfre Aug 3, 2022
720a1c9
Doc: update for preparing release 1.4
ricsanfre Aug 3, 2022
32cab9c
Updating external node metrics service to match node exporter grafana…
ricsanfre Aug 3, 2022
934c17f
Updating 1.4 release announcement
ricsanfre Aug 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The cluster can be re-deployed in minutes as many times as needed for testing ne

## Scope

Automatically deploy and configure a lightweight Kubernetes flavor based on [K3S](https://ks3.io/) and deploy cluster basic services such as: 1) distributed block storage for POD's persistent volumes, [LongHorn](https://longhorn.io/), 2) centralized monitoring tool, [Prometheus](https://prometheus.io/) 3) centralized log management, EFK stack ([Elasticsearch](https://www.elastic.co/elasticsearch/)-[Fluentbit](https://fluentbit.io/)-[Kibana](https://www.elastic.co/kibana/), 3) backup/restore solution for the cluster, [Velero](https://velero.io/) and [Restic](https://restic.net/) and 4) service mesh architecture, [Linkerd](https://linkerd.io/)
Automatically deploy and configure a lightweight Kubernetes flavor based on [K3S](https://ks3.io/) and deploy cluster basic services such as: 1) distributed block storage for POD's persistent volumes, [LongHorn](https://longhorn.io/), 2) centralized monitoring tool, [Prometheus](https://prometheus.io/) 3) centralized log management, EFK stack ([Elasticsearch](https://www.elastic.co/elasticsearch/)-[Fluentd](https://www.fluentd.org/)/[Fluentbit](https://fluentbit.io/)-[Kibana](https://www.elastic.co/kibana/), 4) backup/restore solution for the cluster, [Velero](https://velero.io/) and [Restic](https://restic.net/) and 5) service mesh architecture, [Linkerd](https://linkerd.io/)

The following picture shows the set of opensource solutions used so far in the cluster, which installation process has been documented and its deployment has been automated with Ansible:

Expand All @@ -30,7 +30,7 @@ The following picture shows the set of opensource solutions used so far in the c

## Cluster architecture and hardware

Home lab architecture, showed in the picture bellow, consist of a Kubernetes cluster of 4 nodes (1 master and 3 workers) and a firewall, built with another Raspberry PI, to isolate cluster network from your home network.
Home lab architecture, showed in the picture below, consist of a Kubernetes cluster of 4 nodes (1 master and 3 workers) and a firewall, built with another Raspberry PI, to isolate cluster network from your home network.


<p align="center">
Expand All @@ -51,4 +51,4 @@ Check out the documentation [Quick Start guide](http://picluster.ricsanfre.com/d

## About the Project

This project has been started in June 2021 by Ricardo Sanchez
This project was started in June 2021 by Ricardo Sanchez
10 changes: 7 additions & 3 deletions docs/_data/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,15 @@
- traefik
- certmanager
- longhorn
- title: Cluster Monitoring
- title: Log Management
docs:
- logging
- elasticsearch
- logging-forwarder-aggregator
- title: Monitoring
docs:
- prometheus
- title: Cluster Backup
- title: Backup
docs:
- backup
- title: Service Mesh
Expand All @@ -37,4 +41,4 @@
- k8s-networking
- k8s-commands
- iscsi-commands
- pi-commands
- pi-commands
Loading