-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: add "Architecture and Load Balancers" #2574
base: main
Are you sure you want to change the base?
Changes from 1 commit
e42706b
4a658b5
1365167
fe12c49
de3483c
2c28181
1c06948
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# Architectural Considerations | ||
|
||
Building a syslog ingestion architecture is complex and requires careful planning. The syslog protocol prioritizes speed and efficiency, often at the expense of resiliency and reliability. Due to these trade-offs, traditional scaling methods may not be directly applicable to syslog. | ||
|
||
This document outlines recommended architectural solutions, along with alternative or unsupported methods that some users have found viable. | ||
|
||
## Edge vs. Centralized Collection | ||
|
||
While TCP and TLS are supported, UDP remains the dominant protocol for syslog transport in many data centers. Since syslog is a "send and forget" protocol, it performs poorly when routed through complex network infrastructures, including front-end load balancers and WAN. | ||
|
||
### Recommendation: Use Edge Collection | ||
|
||
The most reliable way to gather syslog traffic is through edge collection rather than centralized collection. If your syslog server is centrally located, UDP and stateless TCP traffic cannot adapt, leading to data loss. | ||
|
||
## Avoid Load Balancing for Syslog | ||
|
||
For optimal performance, scale vertically by fine-tuning a single, robust server. Key tools and methods for enhancing performance on your SC4S server are documented in: | ||
|
||
1. [Fine-tune for TCP](tcp-optimization.md) | ||
2. [Fine-tune for UDP](udp-optimization.md) | ||
|
||
We advise against co-locating syslog-ng servers for horizontal scaling with load balancers. The challenges of load balancing for horizontal scaling are outlined in the [Load Balancer's Overview](lb/index.md) section. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Avoid co-locating syslog-ng servers for horizontal scaling with load balancers. Load balancing challenges for horizontal scaling are described in the Load Balancer's Overview section. |
||
|
||
## High Availability (HA) Considerations | ||
|
||
Syslog, being prone to data loss, can only achieve "mostly available" data collection. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Syslog is prone to data loss and can only achieve "mostly available" data collection. |
||
|
||
### HA Without Load Balancers | ||
|
||
Load balancing does not suit syslog’s stateless, unacknowledged traffic. More data is preserved with simpler designs, such as vMotioned VMs. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Load balancing does not work well with syslog’s stateless, unacknowledged traffic. To preserve more data by using simpler designs, such as vMotioned VMs. |
||
|
||
The optimal deployment model for high availability uses a [Microk8s](https://microk8s.io/) setup with MetalLB in BGP mode. This method implements load balancing through destination network translation, providing better HA results. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The best deployment model for high availability traffic is a Microk8s setup with MetalLB in BGP mode. This implements load balancing through destination network translation, providing better HA results. |
||
|
||
## UDP vs. TCP | ||
|
||
Syslog optimally uses UDP for log forwarding due to its low overhead and simplicity. UDP's streaming nature eliminates the need for network session establishment, which reduces network strain and avoids complex verification processes. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Syslog uses UDP for log forwarding due to its low overhead and simplicity. UDP's streaming nature eliminates the need for network session establishment, which reduces network strain and avoids complex verification processes. |
||
|
||
### Drawbacks of TCP | ||
|
||
While TCP uses acknowledgement signals (ACKS) to mitigate data loss, issues still arise, such as: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. TCP uses acknowledgement signals (ACKS) to mitigate data loss. Issues may still arise, including: |
||
|
||
- Loss of events during TCP session establishment | ||
- Slow acknowledgment signals leading to buffer overflows | ||
- Lost acknowledgments causing closed connections | ||
- Data loss during server restarts | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
### When to Use UDP vs. TCP | ||
|
||
Use UDP by default for syslog forwarding, switching to TCP for larger syslog events that exceed UDP packet limits (common with Web Proxy, DLP, and IDS sources). | ||
|
||
The following resources will help you choose the best protocol for your setup: | ||
|
||
1. [Run performance tests for TCP](performance-tests.md#check-your-tcp-performance) | ||
2. [Run performance tests for UDP](performance-tests.md#check-your-udp-performance) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# Load Balancers Are Not a Best Practice for SC4S | ||
|
||
Be aware of the following issues that may arise from load balancing syslog traffic: | ||
- Load balancing for scale can lead to increased data loss due to normal device operations and buffer overflows. | ||
- Front-side load balancing often results in uneven data distribution on the upstream side. | ||
- The default behavior of Layer 4 (L4) load balancers is to overwrite the client's source IP with their own. Preserving the real source IP requires additional configuration. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
### Recommendations for Using Load Balancers: | ||
- Preserve the actual source IP of the sending device. | ||
- Avoid using load balancers without High Availability (HA) mode. | ||
- TCP/TLS load balancers often do not account for the load on individual connections and may favor one instance over others. Ensure all members in a resource pool are vertically scaled to handle the full workload. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Best practices when using load balancers:
|
||
For **TCP/TLS**, you can use either a DNAT configuration or SNAT with the "PROXY" protocol enabled by setting `SC4S_SOURCE_PROXYCONNECT=yes`. | ||
For **UDP**, traffic can only pass through a load balancer using DNAT. | ||
|
||
This section of the documentation discusses various load balancing solutions and potential configurations, along with known issues. | ||
Please note that load balancing syslog traffic in front of SC4S is not supported by Splunk, and additional support from the load balancer vendor may be required. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,190 @@ | ||
# Nginx Open Source | ||
|
||
This section of the documentation describes the challenges of load balancing syslog traffic using Nginx Open Source. | ||
|
||
There are several key disadvantages to using Nginx Open Source for this purpose: | ||
- Nginx Open Source does not provide active health checking, which is essential for UDP DSR (Direct Server Return) load balancing. | ||
- Even with round-robin load balancing, traffic distribution can often be uneven, leading to overloaded instances in the pool. This results in growing queues, causing delays, data drops, and potential memory or disk issues. | ||
- Without High Availability, an Nginx Open Source load balancer becomes a new single point of failure. | ||
|
||
**Please note that Splunk only supports SC4S**. If issues arise due to the load balancer, please reach out to the Nginx support team. | ||
|
||
## Install Nginx | ||
|
||
1. Refer to the Nginx documentation for instructions on installing Nginx **with the stream module**, which is required for TCP/UDP load balancing. For example, on Ubuntu: | ||
```bash | ||
sudo apt update | ||
sudo apt -y install nginx libnginx-mod-stream | ||
``` | ||
|
||
2. (Optionally) Refer to the Nginx documentation for instructions on fine-tuning Nginx performance. For example, you can update the `events` section in your Nginx configuration file: | ||
|
||
`/etc/nginx/nginx.conf` | ||
```conf | ||
events { | ||
worker_connections 20480; | ||
multi_accept on; | ||
use epoll; | ||
} | ||
``` | ||
Please note that actual load balancer fine-tuning is beyond the scope of the SC4S team's responsibility. | ||
|
||
## Preserving Source IP | ||
The default behavior of Nginx is to overwrite the source IP with the LB's IP. While some users accept this behavior, it is recommended to preserve the original source IP of the message. | ||
|
||
Nginx offers three methods to preserve the source IP: | ||
|
||
| Method | Protocol | | ||
|-----------------------------|------------| | ||
| PROXY protocol | TCP* | | ||
| Transparent IP | TCP/TLS | | ||
| Direct Server Return (DSR) | UDP | | ||
|
||
* TLS PROXY protocol support in SC4S is scheduled for implementation. | ||
|
||
Examples for setting up Nginx with the PROXY protocol and DSR are provided below. The Transparent IP method requires complex network configuration. For more details, refer to [this Nginx blog post](https://www.f5.com/company/blog/nginx/ip-transparency-direct-server-return-nginx-plus-transparent-proxy). | ||
|
||
|
||
## Option 1: Configure Nginx Open Source with the PROXY Protocol | ||
|
||
### Advantages: | ||
- Easy to set up | ||
|
||
### Disadvantages: | ||
- Available only for TCP, not for UDP or TLS | ||
- Overwriting the source IP in SC4S is not ideal; the `SOURCEIP` is a hard macro and only `HOST` can be overwritten | ||
- Overwriting the source IP is available only in SC4S versions greater than 3.4.0 | ||
|
||
### Configuration | ||
|
||
1. On your load balancer (LB) node, add a configuration similar to the following: | ||
`/etc/nginx/modules-enabled/sc4s.conf` | ||
```conf | ||
stream { | ||
# Define upstream for each of SC4S hosts and ports | ||
# Default SC4S TCP ports are 514, 601 | ||
# Include your custom ports if applicable | ||
upstream stream_syslog_514 { | ||
server <SC4S_IP_1>:514; | ||
server <SC4S_IP_2>:514; | ||
} | ||
upstream stream_syslog_601 { | ||
server <SC4S_IP_1>:601; | ||
server <SC4S_IP_2>:601; | ||
} | ||
|
||
# Define a common configuration block for all servers | ||
map $server_port $upstream_name { | ||
514 stream_syslog_514; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably we also need tell that we can use port segregation, it will increase performance:
|
||
601 stream_syslog_601; | ||
} | ||
|
||
# Define a virtual server for each upstream connection | ||
# Ensure 'proxy_protocol' is set to 'on' | ||
server { | ||
listen 514; | ||
listen 601; | ||
mstopa-splunk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
proxy_pass $upstream_name; | ||
|
||
proxy_timeout 3s; | ||
proxy_connect_timeout 3s; | ||
|
||
proxy_protocol on; | ||
} | ||
} | ||
``` | ||
|
||
3. Refer to the Nginx documentation to find the command to reload the service, for example: | ||
```bash | ||
sudo nginx -s reload | ||
``` | ||
|
||
4. Add the following parameter to the SC4S configuration and restart your instances: | ||
`/opt/sc4s/env_file` | ||
```conf | ||
SC4S_SOURCE_PROXYCONNECT=yes | ||
``` | ||
|
||
### Test Your Setup | ||
Send TCP messages to the load balancer and verify that they are correctly received in Splunk with the host set to your source IP, not the LB's IP: | ||
|
||
```bash | ||
# Test message without IETF frame for port 514/TCP: | ||
echo "hello world" | netcat <LB_IP> 514 | ||
# Test message with IETF frame for port 601/TCP: | ||
echo "11 hello world" | netcat <LB_IP> 601 | ||
``` | ||
|
||
3. Run performance tests based on the [Check TCP Performance](tcp_performance_tests.md) section. | ||
|
||
| Receiver | Performance | | ||
|---------------------------|--------------------------------| | ||
| Single SC4S Server | 4,341,000 (71,738.98 msg/sec) | | ||
| Load Balancer + 2 Servers | 5,996,000 (99,089.03 msg/sec) | | ||
|
||
Please note that load balancer fine-tuning is beyond the scope of the SC4S team's responsibility. For assistance in increasing the TCP throughput of your load balancer instance, contact the Nginx support team. | ||
|
||
## Option 2: Configure Nginx with DSR (Direct Server Return) | ||
|
||
### Advantages: | ||
- Works for UDP | ||
- Saves one hop and additional wrapping | ||
|
||
### Disadvantages: | ||
- DSR setup requires active health checks because the load balancer cannot expect responses from the upstream. Active health checks are not available in Nginx Open Source, so switch to Nginx Plus or implement your own active health checking. | ||
- Requires superuser privileges. | ||
- For cloud users, this might require disabling `Source/Destination Checking` (tested with AWS). | ||
|
||
1. In the main Nginx configuration, update the `user` to root: | ||
`/etc/nginx/nginx.conf` | ||
```conf | ||
user root; | ||
``` | ||
|
||
2. Add a configuration similar to the following in: | ||
`/etc/nginx/modules-enabled/sc4s.conf` | ||
```conf | ||
stream { | ||
# Define upstream for each of SC4S hosts and ports | ||
# Default SC4S UDP port is 514 | ||
# Include your custom ports if applicable | ||
upstream stream_syslog_514 { | ||
server <SC4S_IP_1>:514; | ||
server <SC4S_IP_2>:514; | ||
} | ||
|
||
# Define connections to each of your upstreams. | ||
# Ensure to include `proxy_bind` and `proxy_responses 0`. | ||
server { | ||
listen 514 udp; | ||
proxy_pass stream_syslog_514; | ||
|
||
proxy_bind $remote_addr:$remote_port transparent; | ||
proxy_responses 0; | ||
} | ||
} | ||
``` | ||
|
||
3. Refer to the Nginx documentation to find the command to reload the service, for example: | ||
```bash | ||
sudo nginx -s reload | ||
``` | ||
|
||
4. Ensure that you disable `Source/Destination Checking` on your load balancer's host if you are working on AWS. | ||
|
||
### Test Your Setup | ||
1. Send UDP messages to the load balancer and verify that they are correctly received in Splunk with the correct host IP: | ||
```bash | ||
echo "hello world" > /dev/udp/<LB_IP>/514 | ||
``` | ||
|
||
2. Run performance tests | ||
|
||
| Receiver / Drops Rate for EPS (msgs/sec) | 4,500 | 9,000 | 27,000 | 50,000 | 150,000 | 300,000 | | ||
|------------------------------------------|--------|--------|--------|--------|---------|---------| | ||
| Single SC4S Server | 0.33% | 1.24% | 52.31% | 74.71% | -- | -- | | ||
| Load Balancer + 2 Servers | 1% | 1.19% | 6.11% | 47.64% | -- | -- | | ||
| Single Finetuned SC4S Server | 0% | 0% | 0% | 0% | 47.37% | -- | | ||
| Load Balancer + 2 Finetuned Servers | 0.98% | 1.14% | 1.05% | 1.16% | 3.56% | 55.54% | | ||
|
||
Please note that load balancer fine-tuning is beyond the scope of the SC4S team's responsibility. For assistance in minimizing UDP drops on the load balancer side, contact the Nginx support team. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scale vertically by fine-tuning a single, robust server. Tools and methods for enhancing performance on your SC4S server are documented in: