Traceability
This document is to come up with approaches to capture transactional information from requests handled by Kong gateway. Once we capture the required information, we can build our transaction events and send them to Condor, and then we can see the metrics for Kong Services on Amplify Central → API Observer.
1) Using File Log plugin for Kong
- The File log plugin adds request & response data to a log file whose path can be specified by us
- The plugin can be enabled for a particular Kong Service, Route, Consumer or it can be enabled globally
Does anyone use a similar approach?
- Yes, the v7 traceability agent uses the approach wherein the agent monitors the API Gateway log files for traffic and queries API Gateway APIs for additional information to build transaction events and send it to Condor
Advantages:
- Can use v7 traceability agent as a reference for implementation
- If agents fails for any reason, trace logs are persisted so they can be accessed later and push to Amplify Central not loosing any transaction
Any disadvantages?
- Need to keep track of file size
- File I/O is slow
2) Using HTTP Log plugin for Kong - Recommended approach
- The HTTP log plugin sends request and response data to an HTTP server
- This plugin can be enabled for a particular Kong Service, Route, Consumer or it can be enabled globally
Does anyone use a similar approach?
- Yes, Amplify Central SaaS service named Discovery controller uses a similar approach. It exposes an HTTP endpoint that is used by the relay service to send events to it. These events are queued and processed.
Advantages
- Can use Discovery controller as a reference for implementation
- No need to worry about log file maintenance
- Faster compared to using File log plugin, since the file I/O latency is overcome
Disadvantages
- If agents fails for any reason, trace logs will not be pushed to Amplify Central as they are not persisted
3) Extended HTTP Log plugin or extended File Log plugin
- They are just extensions of above mentioned plugins, meaning they log request body and response body as well in addition to other info logged by the above plugins
- Right now the request and response bodies are not useful to us, so no need to take advantage of the extended plugins
4) Using TCP Log plugin for Kong
- The TCP log plugin sends request and response data to a TCP server
- This plugin can be enabled for a particular Kong Service, Route, Consumer or it can be enabled globally
- Using this plugin is not serving any specific purpose for us right now than using HTTP Log plugin
Log format used by both of the above plugins is as shown below:
{
"request": {
"method": "GET",
"uri": "/get",
"url": "http://httpbin.org:8000/get",
"size": "75",
"querystring": {},
"headers": {
"accept": "*/*",
"host": "httpbin.org",
"user-agent": "curl/7.37.1"
},
"tls": {
"version": "TLSv1.2",
"cipher": "ECDHE-RSA-AES256-GCM-SHA384",
"supported_client_ciphers": "ECDHE-RSA-AES256-GCM-SHA384",
"client_verify": "NONE"
}
},
"upstream_uri": "/",
"response": {
"status": 200,
"size": "434",
"headers": {
"Content-Length": "197",
"via": "kong/0.3.0",
"Connection": "close",
"access-control-allow-credentials": "true",
"Content-Type": "application/json",
"server": "nginx",
"access-control-allow-origin": "*"
}
},
"tries": [
{
"state": "next",
"code": 502,
"ip": "127.0.0.1",
"port": 8000
},
{
"ip": "127.0.0.1",
"port": 8000
}
],
"authenticated_entity": {
"consumer_id": "80f74eef-31b8-45d5-c525-ae532297ea8e",
"id": "eaa330c0-4cff-47f5-c79e-b2e4f355207e"
},
"route": {
"created_at": 1521555129,
"hosts": null,
"id": "75818c5f-202d-4b82-a553-6a46e7c9a19e",
"methods": null,
"paths": [
"/example-path"
],
"preserve_host": false,
"protocols": [
"http",
"https"
],
"regex_priority": 0,
"service": {
"id": "0590139e-7481-466c-bcdf-929adcaaf804"
},
"strip_path": true,
"updated_at": 1521555129
},
"service": {
"connect_timeout": 60000,
"created_at": 1521554518,
"host": "example.com",
"id": "0590139e-7481-466c-bcdf-929adcaaf804",
"name": "myservice",
"path": "/",
"port": 80,
"protocol": "http",
"read_timeout": 60000,
"retries": 5,
"updated_at": 1521554518,
"write_timeout": 60000
},
"workspaces": [
{
"id":"b7cac81a-05dc-41f5-b6dc-b87e29b6c3a3",
"name": "default"
}
],
"consumer": {
"username": "demo",
"created_at": 1491847011000,
"id": "35b03bfc-7a5b-4a23-a594-aa350c585fa8"
},
"latencies": {
"proxy": 1430,
"kong": 9,
"request": 1921
},
"client_ip": "127.0.0.1",
"started_at": 1433209822425
}
Using the above info, the traceability agent can build the transaction summary event in the below standard format.
Note: The values for the fields here are just examples, strategies to get the actual values for Kong gateway traffic have been mentioned in front of "=>"
<![CDATA[{
"version":"1.0, => does not change for us
"timestamp":1572543208174, => started_at
"tenantId":"304371132327104", => to be passed to the trace agent
"trcbltPartitionId":"304371132327104", => to be passed to the trace agent
"transactionId":"1NMsDp3i1sfnI@Y0oFvD", => can be got by using Kong plugin https://docs.konghq.com/hub/kong-inc/correlation-id/
"environmentId":"eeeeeeeeeeeeeeeeeeeeeeeeeee00001", => to be passed to the trace agent
"type":"transactionSummary", => does not change for us
"transactionSummary":{
"status":"Success", => can be inferred from response.status
"statusDetail":200, => response.status
"duration":16, => latencies.request
"proxy":{
"id":"e4f485636ae6a619016ae6a681fd0000", => service.id
"name":"API Central", => service.name
"revision":1 => irrelevant to Kong gateway
},
"runtime":{
"id":"e4f485636ae6a619016ae6a684a40006", => irrelevant to Kong gateway
"name":"API Central Runtime" => irrelevant to Kong gateway
},
"entryPoint":{
"type":"http", => service.protocol?
"method":"GET", request.method
"path":"/api/v1/sync", request.uri
"host":"apicentral-axway.apiconsumer.theta.xenon.apicentral-k8s.axwaytest.net" => request.url?
}
}
}]]>