HPCC-30795 Extends Jtrace exporter and configuration support

- Adds Otel dependancy - Exposes otlp-http exporter configuration - Defines jtrace processor and exporter config syntax - Adds support for OTLP HTTP/GRCP support - Adds helm/tracing/readme documentation - Adds sample values block for otlp exporters - Adds ca cert path support Signed-off-by: Rodrigo Pastrana <[email protected]>
hpcc-systems · Nov 16, 2023 · 9128cfc · 9128cfc
1 parent e081c75
commit 9128cfc
Show file tree

Hide file tree

Showing 6 changed files with 173 additions and 20 deletions.
diff --git a/helm/examples/tracing/README.md b/helm/examples/tracing/README.md
@@ -0,0 +1,96 @@
+# HPCC Component Instrumentation and Tracing
+
+The HPCC Platform is instrumented to optionally track distributed work actions (Traces) and their various sub-task (spans) via Open Telemetry tooling. Traces and spans track information vital information which can be reported to services which specialize in processing and visualization of standardized trace/span information for the purpose or monitoring and observability of the health of individual requests processed by the platform.
+
+Tracing of HPCC components is enabled by default, and can be configured to fit the needs of the given deployment.
+
+## Configuration
+All configuration options detailed here are part of the HPCC Systems Helm chart, and apply at the global or component level.
+
+### Tracing cofiguration options
+- disabled - (default: false) disables tracking and reporting of internal traces and spans
+- alwaysCreateGlobalIds - If true, assign newly created global ID to any requests that do not supply one.
+- exporter - Defines The type of exporter in charge of forwarding span data to target back-end
+ - type - (defalt: NONE) "OTLP-HTTP" | "OTLP-GRCP" | "OS" | "NONE"
+  - OTLP-HTTP
+   - endpoint - (default localhost:4318) Specifies the target OTLP-HTTP backend
+   - timeOutSecs - (default 10secs)
+   - consoleDebug - (default false)
+  - OTLP-GRCP
+   - endpoint: (default localhost:4317) The endpoint to export to. By default the OpenTelemetry Collector's default endpoint.
+   - useSslCredentials - By default when false, uses grpc::InsecureChannelCredentials; If true uses sslCredentialsCACertPath
+   - sslCredentialsCACertPath - Path to .pem file to be used for SSL encryption.
+   - timeOutSeconds - (default 10secs) Timeout for grpc deadline
+- processor - Controls span processing style. One by one as available, or in batches.
+ - type - (default: simple) "simple" | "batch"
+
+### Sample configuration
+Below is a sample helm values block directing the HPCC tracing framework to process span information serially, and export the data over OTLP/HTTP protocol to localhost:4318 and output export debug information to console:
+
+```console
+global:
+  tracing:
+    exporter:
+      type: OTLP-HTTP
+      consoleDebug: true
+    processor:
+      type: simple
+```
+### Sample configuration command
+
+Sample helm command deploying an HPCC chart named myTracedHPCC using the hpcc helm repo and providing a the above tracing configuration.
+
+```console
+helm install myTracedHPCC hpcc/hpcc -f otlp-http-collector-default.yaml
+```
+## Tracing information
+HPCC tracing information includes data needed to trace requests as they traverse over distributed components, and detailed information pertaining to important request subtasks in the form of span information. Each trace and all its related spans are assigned unique IDs which follow the Open Telemetry standard.
+
+The start and end of spans are reported to HPCC component logs regardless of any exporter related configuration.
+
+Sample span reported as log event:
+```console
+000000A3 MON EVT 2023-10-10 22:12:23.827 24212 25115 Span start: {"Type":"Server","Name":"propagatedServerSpan","GlobalID":"IncomingUGID","CallerID":"IncomingCID","LocalID":"JDbF4xnv7LSWDV4Eug1SpJ","TraceID":"beca49ca8f3138a2842e5cf21402bfff","SpanID":"4b960b3e4647da3f"}
+
+000000FF MON EVT 2023-10-10 22:12:24.927 24212 25115 Span end: {"Type":"Server","Name":"propagatedServerSpan","GlobalID":"IncomingUGID","CallerID":"IncomingCID","LocalID":"JDbF4xnv7LSWDV4Eug1SpJ","TraceID":"beca49ca8f3138a2842e5cf21402bfff","SpanID":"4b960b3e4647da3f"}
+```
+
+Each log statement denotes the time of the tracing event (start or stop), the span type, name, trace and span id, and any HPCC specific attribute such as legacy GlobalID (if any), HPCC CallerID (if any), LocalID (if any).
+
+Spans exported via exporters will contain more detailed information such as explicit start time, duration, and any other attribute assigned to the span by the component instrumentation.
+
+Sample exported span data:
+```json
+{
+    "Name":"propagatedServerSpan",
+    "TraceId":"beca49ca8f3138a2842e5cf21402bfff",
+    "SpanId":"6225221529c24252",
+    "kind":"Server",
+    "ParentSpanId":"4b960b3e4647da3f",
+    "Start":1696983526105561763,
+    "Duration":1056403,
+    "Description":"",
+    "Status":"Unset",
+    "TraceState":"hpcc=4b960b3e4647da3f",
+    "Attributes":{
+        "hpcc.callerid":"IncomingCID",
+        "hpcc.globalid":"IncomingUGID"
+    },
+    "Events":{
+    },
+    "Links":{
+    },
+    "Resources":{
+        "service.name":"unknown_service",
+        "telemetry.sdk.version":"1.9.1",
+        "telemetry.sdk.name":"opentelemetry",
+        "telemetry.sdk.language":"cpp"
+    },
+    "InstrumentedLibrary":"esp"
+
+}
+```
+
+## Directory Contents
+
+- 'otlp-http-collector-default.yaml' - Sample tracing configuration targeting OTLP/HTTP trace collector
diff --git a/helm/examples/tracing/otlp-grcp-collector-default.yaml b/helm/examples/tracing/otlp-grcp-collector-default.yaml
@@ -0,0 +1,6 @@
+global:
+  tracing:
+    exporter:
+      type: OTLP-GRCP
+      endpoint: "localhost:4317"
+      useSslCredentials: false
diff --git a/helm/examples/tracing/otlp-http-collector-default.yaml b/helm/examples/tracing/otlp-http-collector-default.yaml
@@ -0,0 +1,6 @@
+global:
+  tracing:
+    exporter:
+      type: OTLP-HTTP
+      endpoint: "localhost:4318"
+      consoleDebug: true
diff --git a/helm/hpcc/values.schema.json b/helm/hpcc/values.schema.json
@@ -1115,6 +1115,26 @@
         "alwaysCreateTraceIds": {
           "type": "boolean",
           "description": "If true, components generate trace/span ids to aid in unit of worktracing"
+        },
+        "exporter": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "type": "string",
+              "enum": ["OTLP-HTTP", "OTLP-GRCP", "OS", "NONE"],
+              "description": "The type of exporter in charge of forwarding span data to target back-end"
+            }
+          }
+        },
+        "processor": {
+          "type": "object",
+          "properties": {
+            "type": {
+              "type": "string",
+              "enum": ["batch", "simple"],
+              "description": "Defines the manner in which trace data is processed - in batches, or simple as available"
+            }
+          }
         }
       },
       "additionalProperties": { "type": ["integer", "string", "boolean"] }

diff --git a/system/jlib/CMakeLists.txt b/system/jlib/CMakeLists.txt
@@ -261,7 +261,7 @@ target_link_libraries ( jlib
         #   opentelemetry-cpp::otlp_grpc_log_record_exporter     - Imported target of opentelemetry-cpp::otlp_grpc_log_record_exporter
         #   opentelemetry-cpp::otlp_grpc_metrics_exporter        - Imported target of opentelemetry-cpp::otlp_grpc_metrics_exporter
         #   opentelemetry-cpp::otlp_http_client                  - Imported target of opentelemetry-cpp::otlp_http_client
-        #   opentelemetry-cpp::otlp_http_exporter                - Imported target of opentelemetry-cpp::otlp_http_exporter
+        opentelemetry-cpp::otlp_http_exporter #                  - Imported target of opentelemetry-cpp::otlp_http_exporter
         #   opentelemetry-cpp::otlp_http_log_record_exporter     - Imported target of opentelemetry-cpp::otlp_http_log_record_exporter
         #   opentelemetry-cpp::otlp_http_metric_exporter         - Imported target of opentelemetry-cpp::otlp_http_metric_exporter
         #   opentelemetry-cpp::ostream_metrics_exporter          - Imported target of opentelemetry-cpp::ostream_metrics_exporter

diff --git a/system/jlib/jtrace.cpp b/system/jlib/jtrace.cpp
@@ -34,6 +34,7 @@
 #define ForEach(i)              for((i).first();(i).isValid();(i).next())
 
 #include "opentelemetry/exporters/otlp/otlp_grpc_exporter_factory.h"
+#include "opentelemetry/exporters/otlp/otlp_http_exporter_factory.h"
 #include "opentelemetry/exporters/otlp/otlp_http_exporter_options.h"
 #include "opentelemetry/exporters/memory/in_memory_span_data.h"
 
@@ -782,31 +783,53 @@ class CTraceManager : implements ITraceManager, public CInterface
                         exporter = opentelemetry::exporter::trace::OStreamSpanExporterFactory::Create();
                         DBGLOG("Tracing to stdout/err...");
                     }
-                    else if (stricmp(exportType.str(), "OTLP")==0)
+                    else if (stricmp(exportType.str(), "OTLP")==0 || stricmp(exportType.str(), "OTLP-HTTP")==0)
+                    {
+                        opentelemetry::exporter::otlp::OtlpHttpExporterOptions trace_opts;
+                        const char * endPoint = exportConfig->queryProp("@endpoint");
+                        if (endPoint)
+                            trace_opts.url = endPoint;
+
+                        if (exportConfig->hasProp("@timeOutSecs")) //not sure exactly what this value actually affects
+                            trace_opts.timeout = std::chrono::seconds(exportConfig->getPropInt("@timeOutSecs"));
+
+                        // Whether to print the status of the exporter in the console
+                        trace_opts.console_debug = exportConfig->getPropBool("@consoleDebug", false);
+
+                        exporter  = opentelemetry::exporter::otlp::OtlpHttpExporterFactory::Create(trace_opts);
+                        DBGLOG("Exporting traces via OTLP/HTTP to: (%s)", trace_opts.url.c_str());
+                    }
+                    else if (stricmp(exportType.str(), "OTLP-GRPC")==0)
                     {
                         namespace otlp = opentelemetry::exporter::otlp;
 
                         otlp::OtlpGrpcExporterOptions opts;
-                        StringBuffer endPoint;
-                        exportConfig->getProp("@endpoint", endPoint);
-                        opts.endpoint = endPoint.str();
+
+                        const char * endPoint = exportConfig->queryProp("@endpoint");
+                        if (endPoint)
+                            opts.endpoint = endPoint;
 
                         opts.use_ssl_credentials = exportConfig->getPropBool("@useSslCredentials", false);
 
                         if (opts.use_ssl_credentials)
                         {
-                            StringBuffer sslCACert;
-                            exportConfig->getProp("@sslCredentialsCACcert", sslCACert);
-                            opts.ssl_credentials_cacert_as_string = sslCACert.str();
+                            StringBuffer sslCACertPath;
+                            exportConfig->getProp("@sslCredentialsCACertPath", sslCACertPath);
+                            opts.ssl_credentials_cacert_path = sslCACertPath.str();
                         }
 
+                        if (exportConfig->hasProp("@timeOutSecs")) //grpc deadline timeout in seconds
+                            opts.timeout = std::chrono::seconds(exportConfig->getPropInt("@timeOutSecs"));
+
                         exporter = otlp::OtlpGrpcExporterFactory::Create(opts);
-                        DBGLOG("Tracing to OTLP (%s)", endPoint.str());
+                        DBGLOG("Exporting traces via OTLP/GRPC to: (%s)", opts.endpoint.c_str());
                     }
                     else if (stricmp(exportType.str(), "Prometheus")==0)
                         DBGLOG("Tracing to Prometheus currently not supported");
-                    else if (stricmp(exportType.str(), "HPCC")==0)
-                        DBGLOG("Tracing to HPCC JLog currently not supported");
+                    else if (stricmp(exportType.str(), "NONE")==0)
+                        DBGLOG("Tracing exporter set to 'NONE', no trace exporting will be performed");
+                    else
+                        DBGLOG("Tracing exporter type not supported: '%s', no trace exporting will be performed", exportType.str());
                 }
                 else
                     DBGLOG("Tracing exporter type not specified");
@@ -882,22 +905,24 @@ class CTraceManager : implements ITraceManager, public CInterface
             {
                 const char * simulatedGlobalYaml = R"!!(global:
     tracing:
-        disable: true
+        disabled: false
         exporter:
-          type: OTLP
-          endpoint: "localhost:4317"
-          useSslCredentials: true
-          sslCredentialsCACcert: "ssl-certificate"
+          type: OTLP-HTTP
+          timeOutSecs: 15
+          consoleDebug: true
         processor:
-          type: batch
+          type: simple
     )!!";
                 testTree.setown(createPTreeFromYAMLString(simulatedGlobalYaml, ipt_none, ptr_ignoreWhiteSpace, nullptr));
                 traceConfig = testTree->queryPropTree("global/tracing");
             }
+            if (traceConfig)
+            {
+                StringBuffer xml;
+                toXML(traceConfig, xml);
+                DBGLOG("traceConfig tree: %s", xml.str());
+            }
 
-            StringBuffer xml;
-            toXML(traceConfig, xml);
-            DBGLOG("traceConfig tree: %s", xml.str());
 #endif
             bool disableTracing = traceConfig && traceConfig->getPropBool("@disabled", false);