diff --git a/.github/styles/kong/dictionary.txt b/.github/styles/kong/dictionary.txt index 6d8ecb08862a..67dd50636dbb 100644 --- a/.github/styles/kong/dictionary.txt +++ b/.github/styles/kong/dictionary.txt @@ -195,6 +195,7 @@ Datadog datadog datagram datagrams +datakit dataplane dataplanes db-ip diff --git a/app/_data/docs_nav_gateway_3.9.x.yml b/app/_data/docs_nav_gateway_3.9.x.yml index 2f46f5c56053..5d5c1ee577c5 100644 --- a/app/_data/docs_nav_gateway_3.9.x.yml +++ b/app/_data/docs_nav_gateway_3.9.x.yml @@ -362,6 +362,10 @@ items: items: - text: Overview url: /kong-enterprise/datakit/ + - text: Get Started with Datakit + url: /kong-enterprise/datakit/get-started/ + - text: Datakit Configuration Reference + url: /kong-enterprise/datakit/configuration/ - title: Kong AI Gateway icon: /assets/images/icons/documentation/icn-ai.svg diff --git a/app/_data/installation/gateway.yml b/app/_data/installation/gateway.yml index cb5b83287b33..bbfb98761d86 100644 --- a/app/_data/installation/gateway.yml +++ b/app/_data/installation/gateway.yml @@ -1,7 +1,7 @@ # e.g.: https://cloudsmith.io/~kong/repos/internal-gateway-37/pub-keys/ "38": - rsa_key: E4186B13EAE1A2D5 - gpg_key: 8F87A07D181DAA6B + rsa_key: EE30089B2CC28C9A + gpg_key: B9DCD032B1696A89 "37": rsa_key: A757CBAFE0D65143 gpg_key: C05D9BEAEB9E8E18 diff --git a/app/_src/gateway/kong-enterprise/datakit/configuration.md b/app/_src/gateway/kong-enterprise/datakit/configuration.md new file mode 100644 index 000000000000..133f07053589 --- /dev/null +++ b/app/_src/gateway/kong-enterprise/datakit/configuration.md @@ -0,0 +1,248 @@ +--- +title: Datakit configuration reference +badge: enterprise +alpha: true +--- + +Datakit can be configured via the `/plugins` endpoint. + +## The data model + +The data types are based on those of [serde-json], so the representable value types are: + +* Null +* Boolean +* Number +* String +* Array (a vector of values) +* Object (a map from strings to values) + +## The execution model + +Nodes can have input ports and output ports. +Input ports consume data. Output ports produce data. + +You can link one node's output port to another node's input port. +An input port can receive at most one link, that is, data can only arrive +into an input via one other node. Therefore, there are no race conditions. + +An output port can be linked to multiple nodes. Therefore, one node can +provide data to several other nodes. + +Each node triggers at most once. + +A node only triggers when data is available to all its connected input ports; +that is, only when all nodes connected to its inputs have finished +executing. + +## Node types + +The following node types are implemented: + +**Node type** | **Input ports** | **Output ports** | **Supported attributes** +--------------------:|:--------------------------:|:-----------------:|:----------------------------- +`call` | `body`, `headers`, `query` | `body`, `headers` | `url`, `method`, `timeout` +`jq` | user-defined | user-defined | `jq` +`handlebars` | user-defined | `output` | `template`, `content_type` +`exit` | `body`, `headers` | | `status` +`property` | `value` | `value` | `property`, `content_type` + +### `call` node type + +An HTTP dispatch call. + +#### Input ports + +* `body`: body to use in the dispatch request. +* `headers`: headers to use in the dispatch request. +* `query`: key-value pairs to encode as the query string. + +#### Output ports + +* `body`: body returned as the dispatch response. +* `headers`: headers returned as the dispatch response. +* `error`: triggered if a dispatch error occurs, such as a DNS resolver timeout, etc. + The port returns the error message. + +#### Supported attributes + +* `url` (**required**): the URL to use when dispatching. +* `method`: the HTTP method (default is `GET`). +* `timeout`: the dispatch timeout, in seconds (default is 60). + +### `jq` node type + +Execution of a JQ script for processing JSON. The JQ script is processed +using the [jaq] implementation of the JQ language. + +#### Input ports + +User-defined. Each input port declared by the user will correspond to a +variable in the JQ execution context. A user can declare the name of the port +explicitly, which is the name of the variable. If a port does not have a given +name, it will get a default name based on the peer node and port to which it +is connected, and the name will be normalized into a valid variable name (e.g. +by replacing `.` to `_`). + +#### Output ports + +User-defined. When the JQ script produces a JSON value, that is made available +in the first output port of the node. If the JQ script produces multiple JSON +values, each value will be routed to a separate output port. + +#### Supported attributes + +* `jq`: the JQ script to execute when the node is triggered. + +### `handlebars` node type + +Application of a [Handlebars] template on a raw string, useful for producing +arbitrary non-JSON content types. + +#### Input ports + +User-defined. Each input port declared by the user will correspond to a +variable in the Handlebars execution context. A user can declare the name of +the port explicitly, which is the name of the variable. If a port does not +have a given name, it will get a default name based on the peer node and port +to which it is connected, and the name will be normalized into a valid +variable name (e.g. by replacing `.` to `_`). + +#### Output ports + +* `output`: the rendered template. The output payload will be in raw string + format, unless an alternative `content_type` triggers a conversion. + +#### Supported attributes + +* `template`: the Handlebars template to apply when the node is triggered. +* `content_type`: if set to a MIME type that matches one of DataKit's + supported payload types, such as `application/json`, the output payload will + be converted to that format, making its contents available for further + processing by other nodes (default is `text/plain`, which produces a raw + string). + +### `exit` node type + +Trigger an early exit that produces a direct response, rather than forwarding +a proxied response. + +#### Input ports + +* `body`: body to use in the early-exit response. +* `headers`: headers to use in the early-exit response. + +#### Output ports + +None. + +#### Supported attributes + +* `status`: the HTTP status code to use in the early-exit response (default is + 200). + + +### `property` node type + +Get/set Proxy-Wasm host properties. + +Whether a **get** or **set** operation is performed depends upon the node inputs: + +* If an input port is configured, **set** the property +* If no input port is configured, **get** the property and map it to the output + port + +#### Examples + +Get the current value of `my.property`: + +```yaml +- name: get_property + type: property + property: my.property +``` + +Set the value of `my.property` from `some_other_node.port`: + +```yaml +- name: set_property + type: property + property: my.property + input: some_other_node.port +``` + +Get the value of `my.json-encoded.property` and decode it as JSON: + +```yaml +- name: get_json_property + type: property + property: my.json-encoded.property + content_type: application/json +``` + +#### Input ports + +* `value`: set the property to the value from this port + +#### Output ports + +* `value`: the property value that was retrieved + +#### Supported attributes + +* `property` (**required**): the name of the property +* `content_type`: the MIME type of the property (example: `application/json`) + * **get**: controls how the value is _decoded_ after reading it. + * **set**: controls how the value is _encoded_ before writing it. This is + usually does not need to be specified, as DataKit can typically infer + the correct encoding from the input type. + +## Implicit nodes + +DataKit defines a number of implicit nodes that can be used without being +explicitly declared. These reserved node names cannot be used for user-defined +nodes. These are: + +**Node** | **Input ports** | **Output ports** | **Description** +--------------------:|:--------------------------:|:--------------------------:|:------------------ +`request` | | `body`, `headers`, `query` | the incoming request +`service_request` | `body`, `headers`, `query` | | request sent to the service being proxied to +`service_response` | | `body`, `headers` | response sent by the service being proxied to +`response` | `body`, `headers` | | response to be sent to the incoming request + +The `headers` ports produce and consume maps from header names to their values. +Keys are header names are normalized to lowercase. +Values are strings if there is a single instance of a header, +or arrays of strings if there are multiple instances of the same header. + +The `query` ports produce and consume maps with key-value pairs representing +decoded URL query strings. If the value in the pair is JSON null, +the key is encoded without a value (to encode `key=null`, use `"null"` +as a value). + +The `body` output ports produce either raw strings or JSON objects, +depending on their corresponding `Content-Type` values. + +Likewise, the `body` input ports accept either raw strings or JSON objects, +and both their `Content-Type` and `Content-Length` are automatically adjusted, +according to the type and size of the incoming data. + +## Debugging + +DataKit includes support for debugging your configuration. + +### Execution tracing + +By setting the `X-DataKit-Debug-Trace` header, DataKit records the execution +flow and the values of intermediate nodes, reporting the output in the request +body in JSON format. + +If the debug header value is set to `0`, `false`, or `off`, this is equivalent to +unsetting the debug header: tracing will not happen and execution will run +as normal. Any other value will enable debug tracing. + +--- + +[serde-json]: https://docs.rs/serde_json/latest/serde_json/ +[Handlebars]: https://docs.rs/handlebars/latest/handlebars/ +[jaq]: https://lib.rs/crates/jaq \ No newline at end of file diff --git a/app/_src/gateway/kong-enterprise/datakit/get-started.md b/app/_src/gateway/kong-enterprise/datakit/get-started.md new file mode 100644 index 000000000000..ac1230dd3f4c --- /dev/null +++ b/app/_src/gateway/kong-enterprise/datakit/get-started.md @@ -0,0 +1,134 @@ +--- +title: Get started with Datakit +badge: enterprise +alpha: true +--- + +## Enable WASM filters + +In `kong.conf`, set: + +``` +wasm = on +``` + +Or export the setting via an environment variable: + +```sh +KONG_WASM=on +``` + +Reload or start {{site.base_gateway}} to apply the configuration. + +## Create a service and a route + +{% navtabs %} +{% navtab decK (YAML) %} + +Add the following configuration to a `kong.yaml` file: + +```yaml +_format_version: "3.0" +services: +- url: http://127.0.0.1:8001/ + name: my-service + routes: + - name: my-route + paths: + - / + strip_path: true +``` +{% endnavtab %} +{% navtab Admin API %} + +1. Create a service: + ```bash + curl -i -X POST http://localhost:8001/services \ + --data "name=my-service" \ + --data "url=http://127.0.0.1:8001/" + ``` + +1. Create a route: + ```bash + curl -i -X POST http://localhost:8001/services/my-service/routes \ + --data "name=my-route" \ + --data "paths[]=/" \ + --data "strip_path=true" + ``` + +{% endnavtab %} +{% endnavtabs %} + +## Enable Datakit + +Let's test out Datakit by combining responses from two third party API calls, then returning directly to the client. + +In the following example, replace `SERVICE_NAME|ID` with `my-service`, or with your own service name: + +{% plugin_example %} +plugin: kong-inc/acl +name: datakit +config: + debug: true + nodes: + - name: CAT_FACT + type: call + url: https://catfact.ninja/fact + - name: DOG_FACT + type: call + url: https://dogapi.dog/api/v1/facts + - name: JOIN + type: jq + inputs: + - cat: CAT_FACT.body + - dog: DOG_FACT.body + jq: | + { + "cat_fact": $cat.fact, + "dog_fact": $dog.facts + } + - name: EXIT + type: exit + inputs: + - body: JOIN + status: 200 +targets: + - service +formats: + - curl + - konnect + - yaml + - kubernetes + - terraform +{% endplugin_example %} + +## Validate + +Access the service via the route `my-route` to test Datakit: + +```sh +curl http://locahost:8000/my-route +``` + +You should get a `200` response with a random fact from each fact generator called in the config: + +```json +HTTP/1.1 200 OK +Connection: keep-alive +Content-Length: 432 +Content-Type: application/json +Date: Fri, 06 Dec 2024 18:26:48 GMT +Server: kong/3.9.0.0-enterprise-edition +Via: 1.1 kong/3.9.0.0-enterprise-edition +X-Kong-Proxy-Latency: 744 +X-Kong-Request-Id: 878618103ee7137b7c2f914e107cb454 +X-Kong-Upstream-Latency: 742 + +{ + "cat_fact": "The longest living cat on record according to the Guinness Book belongs to the late Creme Puff of Austin, Texas who lived to the ripe old age of 38 years and 3 days!", + "dog_fact": [ + "Greyhounds can reach a speed of up to 45 miles per hour." + ] +} + +``` \ No newline at end of file diff --git a/app/_src/gateway/kong-enterprise/datakit/index.md b/app/_src/gateway/kong-enterprise/datakit/index.md index 5e20e4b1ff2d..9ee7d303bf0b 100644 --- a/app/_src/gateway/kong-enterprise/datakit/index.md +++ b/app/_src/gateway/kong-enterprise/datakit/index.md @@ -1,26 +1,44 @@ --- title: About Datakit badge: enterprise -badge: tech-preview +alpha: true --- -The {{site.base_gateway}} Datakit is a tool that allows you to interact with third-party APIs. It uses response data from these requests to seed information for subsequent calls, either upstream/back to the client or to other APIs. +The {{site.base_gateway}} Datakit is a tool that allows you to interact with third-party APIs. +It uses response data from these requests to seed information for subsequent calls, either upstream/back to the client or to other APIs. -Datakit is a data flow engine that is built on top of WASM inside {{site.base_gateway}}. +Datakit is a data flow engine built on top of WASM inside {{site.base_gateway}}. +It allows you to create an API workflow, which can include: +* Making calls to third party APIs +* Transforming or combining API responses +* Modifying client requests and service responses +* Adjusting Kong entity configuration +* Returning directly to users instead of proxying + +## Use cases for Datakit + +The following are examples of common use cases for Datakit: + +Use case | Description +---------|------------ +Internal authentication | Use internal auth within your ecosystem by sending response headers to upstreams. +Service augumentation | ? +Dynamic service discovery | ? +Get and set Kong entity properties | Use Datakit to adjust {{site.base_gateway}} entity configurations. For example, you could replace the service URL or set a header on a route. ## How does it work? The core component of Datakit is a node. Nodes are inputs to other nodes, which creates the execution path for a given Datakit configuration. -There are several types of static nodes: -* Third-party HTTP calls -* Transform data and cast variables with `jq` to be shared with other nodes -* Get and set {{site.base_gateway}} configuration data -* Return directly to the client +Datakit provides the following node types: +* [`call`](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/#call-node-type): Third-party HTTP calls +* [`jq`](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/#jq-node-type): Transform data and cast variables with `jq` to be shared with other nodes +* [`handlebars`](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/#handlebars-node-type): Apply a [Handlebars](https://docs.rs/handlebars/latest/handlebars/) template to a raw string, useful for producing arbitrary non-JSON content types +* [`exit`](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/#exit-node-type): Return directly to the client +* [`property`](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/#property-node-type): Get and set {{site.base_gateway}} configuration data -You can also create your own node types. - -Datakit, like any WASM filter, can be applied to services and routes. +Datakit, like any WASM filter, can be applied to services and routes, +You can configure Datakit via the `/plugins` endpoint. The following diagram shows how Datakit can be used to combine two third-party API calls into one response: @@ -39,3 +57,7 @@ sequenceDiagram Datakit->>Client: Uses `jq` to pass both facts in a response {% endmermaid %} +## More information + +* [Get started with Datakit](/gateway/{{page.release}}/kong-enterprise/datakit/get-started/) +* [Datakit configuration reference](/gateway/{{page.release}}/kong-enterprise/datakit/configuration/) \ No newline at end of file