From 77bd78719796e3f7527cdff3ab957709b1a3d510 Mon Sep 17 00:00:00 2001 From: Caio Ramos Casimiro Date: Wed, 15 May 2024 17:20:47 +0100 Subject: [PATCH] [WIP] docs(metrics) --- docs/DIRECTIVES.md | 24 ++++++++++++--- docs/METRICS.md | 77 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 4 deletions(-) create mode 100644 docs/METRICS.md diff --git a/docs/DIRECTIVES.md b/docs/DIRECTIVES.md index 97ae4d9fb..2472efae5 100644 --- a/docs/DIRECTIVES.md +++ b/docs/DIRECTIVES.md @@ -6,6 +6,7 @@ By alphabetical order: - [cache_config](#cache-config) - [compiler](#compiler) - [flag](#flag) +- [max_metric_name_length](#max_metric_name_length) - [module](#module) - [proxy_wasm](#proxy_wasm) - [proxy_wasm_isolation](#proxy_wasm_isolation) @@ -59,6 +60,7 @@ By context: - [tls_verify_cert](#tls_verify_cert) - [tls_verify_host](#tls_verify_host) - `metrics{}` + - [max_metric_name_length](#max_metric_name_length) - [slab_size](#slab_size) - `wasmtime{}` - [cache_config](#cache-config) @@ -534,10 +536,23 @@ shm_queue **usage** | `slab_size ;` ------------:|:---------------------------------------------------------------- **contexts** | `metrics{}` -**default** | `12m` -**example** | `slab_size 5m;` +**default** | `5m` +**example** | `slab_size 12m;` -Set the `size` of the shared memory slab dedicated to metrics storage. +Set the `size` of the shared memory slab dedicated to metrics storage. The value +must be at least 3 * pagesize, e.g. `15k` on Linux. + +> Notes + +The space in memory occupied by a metric depends on its name length, type and +the number of worker processes running. As an example, if all metric names are +64 chars long and 4 workers are running, `5m` can accommodate 20k counters, 20k +gauges, or 4k histograms. + +See the [max_metric_name_legnth](#max_metric_name_length) directive to configure +the max name length in chars for metrics. + +See [Metrics] for a complete description of the metric allocation model. [Back to TOC](#directives) @@ -955,7 +970,8 @@ the `http{}` contexts. [Contexts]: USER.md#contexts [Execution Chain]: USER.md#execution-chain -[SLRU eviction algorithm]: SLRU.md +[Metrics]: METRICS.md [OpenResty]: https://openresty.org/en/ [resolver]: https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver [resolver_timeout]: https://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout +[SLRU eviction algorithm]: SLRU.md diff --git a/docs/METRICS.md b/docs/METRICS.md new file mode 100644 index 000000000..6c020ba46 --- /dev/null +++ b/docs/METRICS.md @@ -0,0 +1,77 @@ +# Metrics + +## Introduction + +In the context of ngx_wasm_module, in accordance with Proxy-Wasm, a metric is +either a counter, a gauge or a histogram. + +A counter is an unsigned 64-bit int that can only be incremented. +A gauge is an unsigned 64-bit int that can take arbitrary values. + +## Histograms + +A histogram, used to represent ranges frequency of a variable, can be defined +as a set of pairs of range and counter. For example, the distribution of +response time of HTTP requests, can be represented as a histogram with ranges +`[0, 16]`, `(16, 128]`, `(128, 1024]` and `(1024, Inf]`. The 1st range's +counter, would be the number of requests whose response time <= 16ms; the 2nd +range's counter, requests whose 16ms < response time <= l28ms; the 3rd range's +counter, requests whose 128ms < response time <= 1024ms; and the last range's +counter, requests whose response time > 1024ms. + +### Binning + +The above example demonstrates a histogram with ranges, or bins, whose upper +bound grows in powers of 2, i.e. 2^4, 2^7 and 2^10. This is usually called +logarithmic binning and is indeed how histograms are implemented in the +ngx_wasm_module. This binning strategy implicates that when a value is recorded, +it's matched with the smalllest upper bound bigger than itself and the paired +counter is incremented. + +### Update and expansion + +Histograms are created with 5 bins, 1 initialized and 4 uninitialized. When a +value is recorded into a histogram, if one of its initialized bins has the upper +bound associated with `v`, then its counter is simply incremented; if not, and +the histogram has uninitialized bins, one of them is initialized with the upper +bound associated with `v` and its counter is incremented; if the histogram's out +of uninitialized bins, it can be expanded, up to 18 bins, to accommodate the +additional bin. If a histogram's reached the limit of bins and it doesn't have +the bin associated with a value, the bin with the smallest upper bound bigger +than `v` have the counter incremented. The bin initialized on histogram creation +has upper bound 2^32 and its counter is incremented if it's the only bin whose +upper bound is bigger than the recorded value. + + +## Memory consumption + +The space in memory occupied by a metric contains its name, value and the +underlying structure representing them in the key-value store. While the +key-value structure has a fixed size of 96 bytes, the sizes of name and value +vary. + +The size in memory of counters and gauges is 8 bytes plus 16 bytes per worker +process. The size of a metric's value grows following the number of workers +because it's segmented across them. Each worker has its own portion of the value +that's only ever updated by itself. When a metric is retrieved, the segments are +consolidated and returned as single metric. This storage strategy allows metric +updates to be performed without the aid of locks at the cost of 16 bytes per +worker. + +Histograms also have a baseline size of 8 bytes, plus 16 bytes per worker. +However, histograms need extra space per worker for bins storage. Each bin takes +8 bytes in memory, so a 5-bin histogram takes 8 bytes plus (16 + 5*8), 56, bytes +per worker. + +As such, in a 4-workers setup, a counter or gauge whose name is 64 chars long +takes 168 bytes, a 5-bin histogram with the same name takes 328 bytes and a +17-bin histogram with the same name takes 712 bytes. + +### Shared memory allocation + +Explain how memory is allocated by ngx shm allocator and how this impacts the +number of metrics that can be represented in memory. + +## Prefixing + +## Nginx Reconfiguration