update schema folder

docs/schema/observability/README.md

@@ -22,6 +22,63 @@ In many occasions, correlation between the logs, traces and metrics is mandatory
 
 For such correlation to be possible the industry has formulated several protocols ([OTEL](https://github.com/open-telemetry), [ECS](https://github.com/elastic/ecs), [OpenMetrics](https://github.com/OpenObservability/OpenMetrics)) for communicating these signals - the Observability schemas.
 
+## Data Correlation
+
+In order to be able to correlate information across different signal (represented in different indices) we introduced the notion of correlation into the schema.
+This information is represented explicitly in both the declarative schema file and the physical mapping file
+
+This information will enable the knowledge to be projected and allow for analytic engine to produce a join query that will take advantage of these relationships.
+The correlation metadata info is exported in the following way:
+
+### Observability Correlation Example:
+
+### Schema related
+In JSON Schema, there is no built-in way to represent relationships directly between multiple schemas, like you would find in a relational database. However, you can establish relationships indirectly by using a combination of `$id`, `$ref`, and consistent property naming across your schemas.
+For example the [`logs.schema`](../../../src/main/resources/schema/observability/logs/logs.schema) file contains the next `$ref` references for the `traceId` & `spanId` fields that belong to the `traces.schema`.
+
+```json5
+  ...
+    "traceId": {
+      "$ref": "https://opensearch.org/schemas/observability/Span#/properties/traceId"
+    },
+    "spanId": {
+      "$ref": "https://opensearch.org/schemas/observability/Span#/properties/spanId"
+    },
+  ...
+```
+
+We can observe that the `traceId` field is defined by referencing to the [Span](../../../src/main/resources/schema/observability/traces/traceGroups.schema) schema and explicitly to the `#/properties/spanId` field reference location.
+
+### Mapping related
+Each mapping template will contain the foreign schemas that are referenced to in that specific mapping file. For example the [`logs.mapping`](../../../src/main/resources/schema/observability/logs/logs.schema) file will contain the next correlation object in the mapping `_meta` section:
+
+```json5
+      "_meta": {
+        "description": "Simple Schema For Observability",
+        "catalog": "observability",
+        "type": "logs",
+        "correlations": [
+          {
+            "field": "spanId",
+            "foreign-schema": "traces",
+            "foreign-field": "spanId"
+          },
+          {
+           "field": "traceId",
+            "foreign-schema": "traces",
+            "foreign-field": "traceId"
+           }
+          ]
+        }
+
+```
+
+Each `correlations` field contains the F.K field name - `spanId` , the referenced schema - `traces` and the source field name in that schema `spanId`
+
+This information can be used to generate the correct join queries on a contextual basis.
+
+![](../../img/correlation.png)
+
 ---
 ## Schema Aware Components
 

docs/schema/observability/logs/README.md

@@ -143,29 +143,29 @@ This field is expected to appear in any future integration or Observability reso
 
 _Inspired by [ECS - http](https://www.elastic.co/guide/en/ecs/current/ecs-http.html), [OTEL - http](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md)_
 
- - `method` - HTTP request method.	`GET; POST; HEAD` (OTEL driven) Correspond with `request.method` (ECS driven)
- - `status_code` - [Http response code](https://tools.ietf.org/html/rfc7231#section-6) (OTEL driven) Correspond with `response.status_code` (ECS driven)
- - `flavor`  - Kind of HTTP protocol used. (OTEL driven) Correspond with `version` (ECS driven)
- - `user_agent` -  Value of the HTTP User-Agent header sent by the client. (OTEL driven)
-
- - `request.id` - A unique identifier for each HTTP request to correlate logs between clients and servers in transactions. (ECS driven)
- - `request_content_length` - The size of the request payload body in bytes. (OTEL driven)  Correspond with `request.bytes` (ECS driven) 
- - `request.body.content` - The full HTTP request body. (ECS driven)
- - `request.referrer` - Referrer for this HTTP request. (ECS driven)
- - `request.header` - HTTP request headers key/value object (OTEL driven)
- - `request.mime_type` - Mime type of the body of the response. (ECS driven)
-
- - `response_content_length` - The size of the response payload body in bytes. (OTEL driven) Correspond with `response.bytes` (ECS driven)
- - `response.body.content` - The full HTTP response body. (ECS driven)
- - `response.header` - HTTP response headers key/value object (OTEL driven)
-
- - `url` - Full HTTP request URL in the form scheme://host[:port]/path?query[#fragment] (OTEL driven)
- - `resend_count` - The ordinal number of request resending attempt (OTEL driven)
-
- - `scheme` - The URI scheme identifying the used protocol. (OTEL driven)
- - `target` - The full request target as passed in a HTTP request line or equivalent. (OTEL driven)
- - `route`  - The matched route (path template in the format used by the respective server framework) (OTEL driven)
- - `client_ip` - The IP address of the original client behind all proxies (OTEL driven) `client.ip` -  IP address of the client (IPv4 or IPv6). (ECS driven)
+- `method` - HTTP request method.	`GET; POST; HEAD` (OTEL driven) Correspond with `request.method` (ECS driven)
+- `status_code` - [Http response code](https://tools.ietf.org/html/rfc7231#section-6) (OTEL driven) Correspond with `response.status_code` (ECS driven)
+- `flavor`  - Kind of HTTP protocol used. (OTEL driven) Correspond with `version` (ECS driven)
+- `user_agent` -  Value of the HTTP User-Agent header sent by the client. (OTEL driven)
+
+- `request.id` - A unique identifier for each HTTP request to correlate logs between clients and servers in transactions. (ECS driven)
+- `request_content_length` - The size of the request payload body in bytes. (OTEL driven)  Correspond with `request.bytes` (ECS driven) 
+- `request.body.content` - The full HTTP request body. (ECS driven)
+- `request.referrer` - Referrer for this HTTP request. (ECS driven)
+- `request.header` - HTTP request headers key/value object (OTEL driven)
+- `request.mime_type` - Mime type of the body of the response. (ECS driven)
+
+- `response_content_length` - The size of the response payload body in bytes. (OTEL driven) Correspond with `response.bytes` (ECS driven)
+- `response.body.content` - The full HTTP response body. (ECS driven)
+- `response.header` - HTTP response headers key/value object (OTEL driven)
+
+- `url` - Full HTTP request URL in the form scheme://host[:port]/path?query[#fragment] (OTEL driven)
+- `resend_count` - The ordinal number of request resending attempt (OTEL driven)
+
+- `scheme` - The URI scheme identifying the used protocol. (OTEL driven)
+- `target` - The full request target as passed in a HTTP request line or equivalent. (OTEL driven)
+- `route`  - The matched route (path template in the format used by the respective server framework) (OTEL driven)
+- `client_ip` - The IP address of the original client behind all proxies (OTEL driven) `client.ip` -  IP address of the client (IPv4 or IPv6). (ECS driven)
 
 #### Communication
 Includes client / server part of the communication

docs/schema/observability/logs/sample/aws/aws_elb-log.json

@@ -0,0 +1,66 @@
+{
+  "@timestamp": "2018-07-02T22:23:00.186Z",
+  "aws": {
+    "elb": {
+      "backend": {
+        "http": {
+          "response": {
+            "status_code": 200
+          }
+        },
+        "ip": "10.0.0.1",
+        "port": "80"
+      },
+      "backend_processing_time": {
+        "sec": 0.001
+      },
+      "matched_rule_priority": "0",
+      "name": "app/my-loadbalancer/50dc6c495c0c9188",
+      "protocol": "http",
+      "request_processing_time": {
+        "sec": 0
+      },
+      "response_processing_time": {
+        "sec": 0
+      },
+      "target_group": {
+        "arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067"
+      },
+      "target_port": [
+        "10.0.0.1:80"
+      ],
+      "target_status_code": [
+        "200"
+      ],
+      "traceId": "Root=1-58337262-36d228ad5d99923122bbe354",
+      "type": "http"
+    }
+  },
+  "cloud": {
+    "provider": "aws"
+  },
+  "http": {
+    "request": {
+      "body": {
+        "bytes": 34
+      },
+      "method": "GET"
+    },
+    "response": {
+      "body": {
+        "bytes": 366
+      },
+      "status_code": 200
+    },
+    "url": "http://www.example.com:80/",
+    "schema": "http"
+  },
+  "communication": {
+    "source": {
+      "address": "192.168.131.39",
+      "ip": "192.168.131.39",
+      "port": 2817
+    }
+  },
+  "traceId": "Root=1-58337262-36d228ad5d99923122bbe354"
+}

docs/schema/observability/logs/sample/cloud.json

@@ -0,0 +1,5 @@
+{
+  "cloud": {
+    "provider": "aws"
+  }
+}

docs/schema/observability/logs/sample/communication.json

@@ -0,0 +1,9 @@
+{
+  "communication": {
+    "source": {
+      "address": "192.168.131.39",
+      "ip": "192.168.131.39",
+      "port": 2817
+    }
+  }
+}

docs/schema/observability/logs/sample/container.json

@@ -0,0 +1,6 @@
+{
+  "container": {
+    "image": "EC2"
+  }
+
+}

docs/schema/observability/logs/sample/http/http.json

@@ -0,0 +1,18 @@
+{
+  "http": {
+    "request": {
+      "body": {
+        "bytes": 34
+      },
+      "method": "GET"
+    },
+    "response": {
+      "body": {
+        "bytes": 366
+      },
+      "status_code": 200
+    },
+    "url": "http://www.example.com:80/",
+    "schema": "http"
+  }
+}

docs/schema/observability/traces/samples/load_samples.md

@@ -5,11 +5,31 @@ For loading the given samples run the next request once the Opensearch cluster i
 `PUT sso_traces-default-namespace/_bulk`
 ```json
 { "create":{ } }
-{"traceId":"4fa04f117be100f476b175e41096e736","spanId":"e275ac9d21929e9b","traceState":[],"parentSpanId":"","name":"client_checkout","kind":"INTERNAL","@timestamp":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"resource":{"telemetry@sdk@name":"opentelemetry","telemetry@sdk@language":"python","telemetry@sdk@version":"0.14b0","service@name":"frontend-client","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"},"status":{"code":0}}
+{"traceId":"4fa04f117be100f476b175e41096e736","spanId":"e275ac9d21929e9b","traceState":[],"parentSpanId":"","name":"client_checkout","kind":"INTERNAL","@timestamp":"2021-11-13T20:20:39+00:00","startTime":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"resource":{"telemetry@sdk@name":"opentelemetry","telemetry@sdk@language":"python","telemetry@sdk@version":"0.14b0","service@name":"frontend-client","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"},"status":{"code":0},"attributes": {"serviceName":"frontend"}}
 { "create":{ } }
-{"traceId":"15d30e4d211d79e10fcaeab97015c90d","spanId":"5bcca8ba513bb54a","traceState":[],"parentSpanId":"","name":"mysql","kind":"CLIENT","@timestamp":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","events":[{"@timestamp":"2021-03-25T17:21:03.044+00:00","name":"exception","attributes":{"exception@message":"1050 %2842S01%29: Table %27User_Carts%27 already exists","exception@type":"ProgrammingError","exception@stacktrace":"Traceback %28most recent call last :File /usr/lib/python3.6/site-packages/opentelemetry/sdk/trace/__init__.py, line 804, in use_span yield spanFile /usr/lib/python3.6/site-packages/opentelemetry/instrumentation/dbapi/__init__.py, line 354, in traced_executionraise exFile /usr/lib/python3.6/site-packages/opentelemetry/instrumentation/dbapi/__init__.py, line 345, in traced_executionresult = query_method%28%2Aargs, %2A%2Akwargs%29File /usr/lib/python3.6/site-packages/mysql/connector/cursor.py"},"droppedAttributesCount":0}],"links":[],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"status":{"message":"1050 %2842S01%29: Table %27User_Carts%27 already exists","code":2},"attributes":{"data_stream":{"type":"span","dataset":"mysql"},"component":"mysql","db@user":"root","net@peer@name":"localhost","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `User_Carts` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"},"resource":{"telemetry@sdk@language":"python","service@name":"database","telemetry@sdk@version":"0.14b0","service@instance@id":"140307275923408","telemetry@sdk@name":"opentelemetry","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"}}
+{"traceId":"15d30e4d211d79e10fcaeab97015c90d","spanId":"5bcca8ba513bb54a","traceState":[],"parentSpanId":"","name":"mysql","kind":"CLIENT","@timestamp":"2021-11-13T20:20:39+00:00","startTime":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","events":[{"@timestamp":"2021-03-25T17:21:03.044+00:00","name":"exception","attributes":{"exception@message":"1050 %2842S01%29: Table %27User_Carts%27 already exists","exception@type":"ProgrammingError","exception@stacktrace":"Traceback %28most recent call last :File /usr/lib/python3.6/site-packages/opentelemetry/sdk/trace/__init__.py, line 804, in use_span yield spanFile /usr/lib/python3.6/site-packages/opentelemetry/instrumentation/dbapi/__init__.py, line 354, in traced_executionraise exFile /usr/lib/python3.6/site-packages/opentelemetry/instrumentation/dbapi/__init__.py, line 345, in traced_executionresult = query_method%28%2Aargs, %2A%2Akwargs%29File /usr/lib/python3.6/site-packages/mysql/connector/cursor.py"},"droppedAttributesCount":0}],"links":[],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"status":{"message":"1050 %2842S01%29: Table %27User_Carts%27 already exists","code":2},"attributes":{"serviceName":"database","data_stream":{"type":"span","dataset":"mysql"},"component":"mysql","db@user":"root","net@peer@name":"localhost","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `User_Carts` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"},"resource":{"telemetry@sdk@language":"python","service@name":"mysql","telemetry@sdk@version":"0.14b0","service@instance@id":"140307275923408","telemetry@sdk@name":"opentelemetry","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"}}
 { "create":{ } }
-{"traceId":"c1d985bd02e1dbb85b444011f19a1ecc","spanId":"55a698828fe06a42","traceState":[],"parentSpanId":"","name":"mysql","kind":"CLIENT","@timestamp":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","events":[{"@timestamp":"2021-03-25T17:21:03+00:00","name":"exception","attributes":{"exception@message":"1050 %2842S01%29: Table  Inventory_Items  already exists","exception@type":"ProgrammingError","exception@stacktrace":"Traceback most recent call last"},"droppedAttributesCount":0}],"links":[{"traceId":"c1d985bd02e1dbb85b444011f19a1ecc","spanId":"55a698828fe06a42w2","traceState":[],"attributes":{"db@user":"root","net@peer@name":"localhost","component":"mysql","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `Inventory_Items` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"},"droppedAttributesCount":0}],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"resource":{"telemetry@sdk@language":"python","telemetry@sdk@version":"0.14b0","service@instance@id":"140307275923408","service@name":"database","telemetry@sdk@name":"opentelemetry","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"},"status":{"code":2,"message":"1050 %2842S01%29: Table %27Inventory_Items%27 already exists"},"attributes":{"data_stream":{"type":"span","namespace":"exceptions","dataset":"mysql"},"db@user":"root","net@peer@name":"localhost","component":"mysql","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `Inventory_Items` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"}}
+{"traceId":"c1d985bd02e1dbb85b444011f19a1ecc","spanId":"55a698828fe06a42","traceState":[],"parentSpanId":"","name":"mysql","kind":"CLIENT","@timestamp":"2021-11-13T20:20:39+00:00","startTime":"2021-11-13T20:20:39+00:00","endTime":"2021-11-14T20:10:41+00:00","events":[{"@timestamp":"2021-03-25T17:21:03+00:00","name":"exception","attributes":{"exception@message":"1050 %2842S01%29: Table  Inventory_Items  already exists","exception@type":"ProgrammingError","exception@stacktrace":"Traceback most recent call last"},"droppedAttributesCount":0}],"links":[{"traceId":"c1d985bd02e1dbb85b444011f19a1ecc","spanId":"55a698828fe06a42w2","traceState":[],"attributes":{"db@user":"root","net@peer@name":"localhost","component":"mysql","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `Inventory_Items` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"},"droppedAttributesCount":0}],"droppedAttributesCount":0,"droppedEventsCount":0,"droppedLinksCount":0,"resource":{"telemetry@sdk@language":"python","telemetry@sdk@version":"0.14b0","service@instance@id":"140307275923408","service@name":"database","telemetry@sdk@name":"opentelemetry","host@hostname":"ip-172-31-10-8.us-west-2.compute.internal"},"status":{"code":2,"message":"1050 %2842S01%29: Table %27Inventory_Items%27 already exists"},"attributes":{"serviceName":"database","data_stream":{"type":"span","namespace":"exceptions","dataset":"mysql"},"db@user":"root","net@peer@name":"localhost","component":"mysql","db@type":"sql","net@peer@port":3306,"db@instance":"","db@statement":"CREATE TABLE `Inventory_Items` %28  `ItemId` varchar%2816%29 NOT NULL,  `TotalQty` int%2811%29 NOT NULL,  PRIMARY KEY %28`ItemId`%29%29 ENGINE=InnoDB"}}
+```
+
+`PUT sso_services_default-namespace/_bulk`
+```json
+{ "create":{ } }
+{"serviceName":"customer","kind":"SPAN_KIND_SERVER","destination":{"resource":"SQL SELECT","domain":"mysql"},"target":null,"traceGroupName":"HTTP GET /dispatch","hashId":"OP/8YTM/rui5D131Dyl3uw=="}
+{ "create":{ } }
+{"serviceName":"openSearch","kind":"SPAN_KIND_CLIENT","destination":null,"target":{"resource":"OpenSearch","domain":"openSearch"},"traceGroupName":"HTTP GET /dispatch","hashId":"NI5NKDfGj0WxtmvIkr7cZQ=="}
+{ "create":{ } }
+{"serviceName":"customer","kind":"SPAN_KIND_SERVER","destination":null,"target":{"resource":"HTTP GET /customer","domain":"customer"},"traceGroupName":"HTTP GET /dispatch","hashId":"4sQ0k4k7V5vsvCf1iJzdqQ=="}
+{ "create":{ } }
+{"serviceName":"search","kind":"SPAN_KIND_SERVER","destination":{"resource":"OpenSearch","domain":"openSearch"},"target":null,"traceGroupName":"HTTP GET /dispatch","hashId":"wfGk6I4tPfRuvTNYkP9dFA=="}
+{ "create":{ } }
+{"serviceName":"database","kind":"SPAN_KIND_CLIENT","destination":null,"target":{"resource":"SQL SELECT","domain":"mysql"},"traceGroupName":"HTTP GET /dispatch","hashId":"hifEI5Vndn1Hrcttnbf0Ig=="}
+{ "create":{ } }
+{"serviceName":"frontend","kind":"SPAN_KIND_CLIENT","destination":{"resource":"HTTP GET /customer","domain":"customer"},"target":null,"traceGroupName":"HTTP GET /dispatch","hashId":"u2t8FF1YHF4t/5Qa68XINw=="}
+{ "create":{ } }
+{"serviceName":"search","kind":"SPAN_KIND_SERVER","destination":null,"target":{"resource":"HTTP GET /search","domain":"search"},"traceGroupName":"HTTP GET /dispatch","hashId":"Zg3QONUPtZIHzS7cN1Yo7Q=="}
+{ "create":{ } }
+{"serviceName":"frontend","kind":"SPAN_KIND_CLIENT","destination":{"resource":"HTTP GET /search","domain":"search"},"target":null,"traceGroupName":"HTTP GET /dispatch","hashId":"AhcxPYfbDX42HAywX7kimQ=="}
 ```
 
 Run the next query to get the Spans kind CLIENT:
@@ -25,4 +45,19 @@ Run the next query to get the Spans kind CLIENT:
     }
   }
 }
+```
+
+Run the next query to get the services by name :
+
+- `GET sso_services-default-namespace/_search`
+```json
+{
+  "query":{
+    "term": {
+      "serviceName":{
+        "value":"customer"
+      }
+    }
+  }
+}
 ```