Improved dp_conf_generate.py

ironcore-dev · Sep 28, 2023 · 8f4eaa5 · 8f4eaa5
1 parent e83c06c
commit 8f4eaa5
Show file tree

Hide file tree

Showing 37 changed files with 959 additions and 579 deletions.
diff --git a/docs/concepts/conf.md b/docs/concepts/conf.md
@@ -0,0 +1,112 @@
+# Config file and command-line arguments
+dp-service supports a custom configuration file that reflects its custom command-line arguments, not EAL ones.
+
+In order to make changes to these easier, along with creating a documentation, argument definition for `getopt` and config file parser is delegated to a python script `hack/dp_conf_generate.py`. You only need to provide a command-line options specification via a JSON file, e.g. `hack/dp_conf.json` and then run `hack/dp_conf_generate.py -s hack/dp_conf.json` to update all needed definitions.
+
+This document will try to explain most of it, but it can be easier to just look at the generated files if you already have experience with `getopt_long()`.
+
+## Compiling the parser
+After running `hack/dp_conf_generate.py`, you need to include the generated C source file into your own source code. While highly irregular, this removes unnecessary linking problems and hides implementation.
+
+The generated header file is to be included where needed, i.e. either directly in a single source file, or in a guarded custom header file as usual.
+
+## Calling the parser
+The only thing needed is to pass `argc` and `argv` to `dp_conf_parse_args()`:
+```
+switch (dp_conf_parse_args(argc, argv)) {
+case DP_CONF_RUNMODE_ERROR:
+	return EXIT_FAILURE;
+case DP_CONF_RUNMODE_EXIT:
+	return EXIT_SUCCESS;
+case DP_CONF_RUNMODE_NORMAL:
+	break;
+}
+```
+The return value is documented in the generated header file.
+
+## JSON Specification
+First, you need to specify the output to generate into:
+```json
+  "header": "../include/dp_conf_opts.h",
+  "source": "../src/dp_conf_opts.c",
+  "markdown": "../docs/deployment/help_dpservice-bin.md",
+```
+And provide a list of options that your program needs to use, e.g. for `--my-option xxx` that takes a string argument and is only used when `#define EXTRA_OPTIONS` is present:
+```json
+  "options": [
+    {
+      "shopt": "m",
+      "lgopt": "my-option",
+      "arg": "MY_ARGUMENT",
+      "help": "this is an example option -m, --my-option MY_ARGUMENT",
+      "var": "storage_variable",
+      "type": "char",
+      "array_size": "DEFINED_STRBUF_LEN",
+	  "ifdef": "EXTRA_OPTIONS"
+    },
+    ...
+  ]
+```
+You can also use `int` arguments:
+```json
+  "options": [
+    {
+      "lgopt": "timeout",
+      "arg": "SECONDS",
+      "help": "this is an example option --timeout SECONDS",
+      "var": "timeout",
+      "type": "int",
+	  "min": 1,
+	  "max": 120,
+	  "default": 60
+    },
+    ...
+  ]
+```
+Or even `enum`s:
+```json
+  "options": [
+    {
+      "shopt": "l",
+      "arg": "LOGLEVEL",
+      "help": "this is an example option -l LOGLEVEL",
+      "var": "log_level",
+      "type": "enum",
+	  "choices": [ "err", "warn", "info" ],
+      "default": "warn"
+    },
+    ...
+  ]
+```
+There are also `bool` options, the default value dictates, what the option will do, i.e. using the option will switch the value to the negative of the default value:
+```json
+  "options": [
+    {
+      "shopt": "i",
+	  "lgopt": "inverse",
+      "help": "invert something",
+      "var": "inverted",
+      "type": "bool",
+      "default": "false"
+    },
+    ...
+  ]
+```
+All these options are parsed automatically. If you however need a special combination of settings, `dp_conf_generate.py` will create a static function signature that forces you (via a compilation error) to implement the parser yourself:
+```json
+  "options": [
+    {
+      "shopt": "x",
+      "arg": "SPECIAL",
+      "help": "this does something complicated"
+    },
+    ...
+  ]
+```
+This will cause a following error:
+```
+In file included from main.c:19:
+opts.c:60:13: error: ‘dp_argparse_opt_x’ used but never defined [-Werror]
+   19 | static void dp_argparse_x(const char *arg);
+```
+Simply implement the function above in the file that included the generated source code.
diff --git a/docs/deployment/commandline.md b/docs/deployment/commandline.md
@@ -1,35 +1,11 @@
 # Dataplane Service Command-line Options
-> This file has been generated by dp_conf_generate.py. As such it should fully reflect the current `dp_service` argument parser.
-
 `dp_service` accepts two sets of options separated by `--`. The first set contains DPDK options, the second `dp_service` options proper. Both sets support `--help`
 
 ## EAL Options
 For more information on EAL options, please see [the official docs](https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html)
 
 ## Dataplane Service Options
-| Option | Argument | Description | Choices |
-|--------|----------|-------------|---------|
-| -h, --help | None | display this help and exit |  |
-| -v, --version | None | display version and exit |  |
-| --pf0 | IFNAME | first physical interface (e.g. eth0) |  |
-| --pf1 | IFNAME | second physical interface (e.g. eth1) |  |
-| --ipv6 | ADDR6 | IPv6 underlay address |  |
-| --vf-pattern | PATTERN | virtual interface name pattern (e.g. 'eth1vf') |  |
-| --dhcp-mtu | SIZE | set the mtu field in DHCP responses (68 - 1500) |  |
-| --dhcp-dns | IPv4 | set the domain name server field in DHCP responses (can be used multiple times) |  |
-| --udp-virtsvc | IPv4,port,IPv6,port | map a VM-accessible IPv4 endpoint to an outside IPv6 UDP service |  |
-| --tcp-virtsvc | IPv4,port,IPv6,port | map a VM-accessible IPv4 endpoint to an outside IPv6 TCP service |  |
-| --wcmp-fraction | FRACTION | weighted-cost-multipath coefficient for pf0 (0.0 - 1.0) |  |
-| --nic-type | NICTYPE | NIC type to use | 'hw' (default) or 'tap' |
-| --no-stats | None | do not print periodic statistics to stdout |  |
-| --no-conntrack | None | disable connection tracking |  |
-| --enable-ipv6-overlay | None | enable IPv6 overlay addresses |  |
-| --no-offload | None | disable traffic offloading |  |
-| --graphtrace-loglevel | LEVEL | verbosity level of packet traversing the graph framework |  |
-| --color | MODE | output colorization mode | 'never' (default), 'always' or 'auto' |
-| --log-format | FORMAT | set the format of individual log lines (on standard output) | 'text' (default) or 'json' |
-| --grpc-port | PORT | listen for gRPC clients on this port |  |
-| --flow-timeout | SECONDS | inactive flow timeout (except TCP established flows) |  |
+All options are described in `dpservice-bin --help`, see [the markdown version of it](help_dpservice-bin.md)
 
 ## Configuration file
 Unless an environment variable `DP_CONF` is set to override the path, `dp_service` uses `/tmp/dp_service.conf` to read configuration before processing any arguments.

diff --git a/docs/deployment/dpservice-dump.md b/docs/deployment/dpservice-dump.md
@@ -0,0 +1,16 @@
+# Dataplane Service Packet Dump Utility
+`dpservice-dump` is a tool to see packets going through dp-service packet processing.
+
+## Command-line Options
+All options are described in `dpservice-dump --help`, see [the markdown version of it](help_dpservice-dump.md)
+
+## Disclaimer
+As this tool attaches to a live packet-processing dp-service, it can cause performance degradation in packet-processing.
+
+Always make sure that the tool detaches cleanly (i.e. prints out `Graphtrace successfully disabled in dp-service`. If this does not happen (or the user is unable to verify), make sure to call `dpservice-dump --stop` to perform a manual clean-up.
+
+## Examples
+`dpservice-dump` prints all ingress/egress packets processed by dp-service.
+`dpservice-dump --drops` also prints dropped packets.
+`dpservice-dump --nodes` also prints packets as they are [going through the graph](../concepts/graphtrace.md)
+
diff --git a/docs/deployment/help_dpservice-bin.md b/docs/deployment/help_dpservice-bin.md
@@ -0,0 +1,28 @@
+# Command-line Options
+
+| Option | Argument | Description | Choices |
+|--------|----------|-------------|---------|
+| -h, --help | None | display this help and exit |  |
+| -v, --version | None | display version and exit |  |
+| --pf0 | IFNAME | first physical interface (e.g. eth0) |  |
+| --pf1 | IFNAME | second physical interface (e.g. eth1) |  |
+| --ipv6 | ADDR6 | IPv6 underlay address |  |
+| --vf-pattern | PATTERN | virtual interface name pattern (e.g. 'eth1vf') |  |
+| --dhcp-mtu | SIZE | set the mtu field in DHCP responses (68 - 1500) |  |
+| --dhcp-dns | IPv4 | set the domain name server field in DHCP responses (can be used multiple times) |  |
+| --udp-virtsvc | IPv4,port,IPv6,port | map a VM-accessible IPv4 endpoint to an outside IPv6 UDP service |  |
+| --tcp-virtsvc | IPv4,port,IPv6,port | map a VM-accessible IPv4 endpoint to an outside IPv6 TCP service |  |
+| --wcmp | PERCENTAGE | weighted-cost-multipath percentage for pf0 (0 - 100) |  |
+| --nic-type | NICTYPE | NIC type to use | 'hw' (default) or 'tap' |
+| --no-stats | None | do not print periodic statistics to stdout |  |
+| --no-conntrack | None | disable connection tracking |  |
+| --enable-ipv6-overlay | None | enable IPv6 overlay addresses |  |
+| --no-offload | None | disable traffic offloading |  |
+| --graphtrace-loglevel | LEVEL | verbosity level of packet traversing the graph framework |  |
+| --color | MODE | output colorization mode | 'never' (default), 'always' or 'auto' |
+| --log-format | FORMAT | set the format of individual log lines (on standard output) | 'text' (default) or 'json' |
+| --grpc-port | PORT | listen for gRPC clients on this port |  |
+| --flow-timeout | SECONDS | inactive flow timeout (except TCP established flows) |  |
+
+> This file has been generated by dp_conf_generate.py. As such it should fully reflect the output of `--help`.
+
diff --git a/docs/deployment/help_dpservice-dump.md b/docs/deployment/help_dpservice-dump.md
@@ -0,0 +1,13 @@
+# Command-line Options
+
+| Option | Argument | Description | Choices |
+|--------|----------|-------------|---------|
+| -h, --help | None | display this help and exit |  |
+| -v, --version | None | display version and exit |  |
+| --drops | None | show dropped packets |  |
+| --nodes | None | show graph node traversal |  |
+| --hw | None | capture offloaded packets (only outgoing VF->PF packets supported) |  |
+| --stop | None | do nothing, only make sure tracing is disabled in dp-service |  |
+
+> This file has been generated by dp_conf_generate.py. As such it should fully reflect the output of `--help`.
+
diff --git a/docs/development/conf.md b/docs/development/conf.md