[8.x] [HTTP/OAS] Add Kibana OpenAPI bundling documentation (#195645) (#…

…197727)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[HTTP/OAS] Add Kibana OpenAPI bundling documentation
(#195645)](#195645)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-24T20:38:22Z","message":"[HTTP/OAS]
Add Kibana OpenAPI bundling documentation (#195645)\n\n**Epic:**
elastic/security-team#9401
(internal)\r\n\r\n## Summary\r\n\r\nThis PR adds Kibana OpenAPI bundling
documentation. The functionality includes multiple scripts and
automation scattered throughout the vast Kibana repo. The goal is to
document the whole chain and make it transparent for the
readers.","sha":"f0c956e58540eb7a0704eeeb1ba01034eae2b8de","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","Team:Core","release_note:skip","v9.0.0","Team:
SecuritySolution","backport:prev-minor","v8.16.0"],"title":"[HTTP/OAS]
Add Kibana OpenAPI bundling
documentation","number":195645,"url":"https://github.com/elastic/kibana/pull/195645","mergeCommit":{"message":"[HTTP/OAS]
Add Kibana OpenAPI bundling documentation (#195645)\n\n**Epic:**
elastic/security-team#9401
(internal)\r\n\r\n## Summary\r\n\r\nThis PR adds Kibana OpenAPI bundling
documentation. The functionality includes multiple scripts and
automation scattered throughout the vast Kibana repo. The goal is to
document the whole chain and make it transparent for the
readers.","sha":"f0c956e58540eb7a0704eeeb1ba01034eae2b8de"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195645","number":195645,"mergeCommit":{"message":"[HTTP/OAS]
Add Kibana OpenAPI bundling documentation (#195645)\n\n**Epic:**
elastic/security-team#9401
(internal)\r\n\r\n## Summary\r\n\r\nThis PR adds Kibana OpenAPI bundling
documentation. The functionality includes multiple scripts and
automation scattered throughout the vast Kibana repo. The goal is to
document the whole chain and make it transparent for the
readers.","sha":"f0c956e58540eb7a0704eeeb1ba01034eae2b8de"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Maxim Palenov <[email protected]>

oas_docs/README.md

@@ -1,9 +1,52 @@
-The `bundle.json` and `bundle.serverless.json` files are generated automatically.
-See `node scripts/capture_oas_snapshot --help` for more info.
+# Kibana API reference documentation
 
-The `output/kibana.serverless.yaml` and `output/kibana.yaml` files join some manually-maintained files with the automatically generated files.
-To add integrate more files into this bundle, edit the appropriate `oas_docs/scripts/merge*.js` files.
-To generate the bundled files, run `make api-docs` (or `make api-docs-serverless` and `make api-docs-stateful`).
-To lint them, run `make api-docs-lint` (or `make api-docs-lint-serverless` and `make api-lint-stateful`).
+Documentation about our OpenAPI bundling workflow and configuration. See Kibana's hosted [stateful](https://www.elastic.co/docs/api/doc/kibana) and [serverless](https://www.elastic.co/docs/api/doc/serverless) docs.
 
-To apply some overlays that perform some post-processing and append some content, run `make api-docs-overlay`.
+## Workflow
+
+The final goal of this workflow is to produce an OpenAPI bundle containing all Kibana's public APIs.
+
+### Step 0
+
+OAS from Kibana's APIs are continuously extracted and captured in `bundle.json` and `bundle.serverless.json` as fully formed OAS documentation. See `node scripts/capture_oas_snapshot --help` for more info.
+
+These bundles form the basis of our OpenAPI bundles to which we append and layer extra information before publishing.
+
+### Step 1
+
+Append pre-existing bundles not extracted from code using [`kbn-openapi-bundler`](../packages/kbn-openapi-bundler/README.md) to produce the final resulting bundles.
+
+To add more files into the final bundle, edit the appropriate `oas_docs/scripts/merge*.js` files.
+
+### Step 2
+
+Apply any final overalys to the document that might include examples or final tweaks (see the ["Scripts"](#scripts) section for more details).
+
+## Scripts
+
+The `oas_docs/scripts` folder contains scripts that point to the source domain-specific OpenAPI bundles and specify additional parameters for producing the final output bundle. Currently, there are the following scripts:
+
+- `merge_ess_oas.js` script produces production an output bundle for ESS
+
+- `merge_serverless_oas.js` script produces production an output bundle for Serverless
+
+### Output Kibana OpenAPI bundles
+
+The `oas_docs/output` folder contains the final resulting Kibana OpenAPI bundles
+
+- `kibana.yaml` production ready ESS OpenAPI bundle
+- `kibana.serverless.yaml` production ready Serverless OpenAPI bundle
+
+## Bundling commands
+
+Besides the scripts in the `oas_docs/scripts` folder, there is an `oas_docs/makefile` to simplify the workflow. The following makefile targets are available:
+
+| Command                    | Description                                                                                                                                                                                                        |
+| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `api-docs`                 | Builds ESS Kibana OpenAPI bundle                                                                                                                                                                                   |
+| `api-docs-serverless`      | Builds Serverless Kibana OpenAPI bundle                                                                                                                                                                            |
+| `api-docs-lint`            | Lints built result bundles                                                                                                                                                                                         |
+| `api-docs-lint-errs`       | Lints built result bundles for errors                                                                                                                                                                              |
+| `api-docs-preview`         | Generates (ESS + Serverless) Kibana OpenAPI bundles preview                                                                                                                                                        |
+| `api-docs-overlay`         | Applies [overlays](https://docs.bump.sh/help/specification-support/overlays/) from `overlays` folder to the Kibana OpenAPI bundles and generate `*.new.yaml` files. Overlays help to fine tune the result bundles. |
+| `api-docs-overlay-preview` | Generates a preview for bundles produced by `api-docs-overlay`                                                                                                                                                     |

x-pack/plugins/security_solution/docs/openapi/README.md

@@ -0,0 +1,66 @@
+# Security Solution API reference documentation
+
+Documentation about Security Solution OpenAPI bundling workflow and configuration. See [Kibana wide docs](../../../../../oas_docs/README.md) for general information.
+
+## Workflow
+
+Security Solution uses **specification first approach**. It means we define OpenAPI spec files describing individual API endpoints (also known as source OpenAPI specs) at first. After that we use tooling in particular [`kbn-openapi-bundler`](../../../../../packages/kbn-openapi-bundler/README.md) to process source OpenAPI specs to produce domain OpenAPI bundles.
+
+The workflow consists of multiple steps and visualized below
+
+![workflow diagram](workflow.png)
+
+This document describes **step 0** implemented on Security Solution's side.
+
+#### Bundling automation (CI integration)
+
+Bundling Security Solution domain OpenAPI bundles is **Step 0** of the workflow. To keep the domain OpenAPI bundles always up-to-date and in sync with the source OpenAPI specs, the bundling runs as part of the `Checks` step in CI on every PR build and on merge builds. If there are any changes to the source OpenAPI files, these changes get propagated to the domain OpenAPI bundles and CI commits the changes. In that case the build is marked as failed and needs to be restarted.
+
+### API Domains
+
+Security Solution has multiple API domains scattered across Kibana. Currently the following API domains are handled in the workflow:
+
+- Security AI Assistant
+
+  - Bundling script: `x-pack/packages/kbn-elastic-assistant-common/scripts/openapi/bundle.js`
+  - Bundles location: `x-pack/packages/kbn-elastic-assistant-common/docs/openapi/{ess|serverless}`
+
+- Security Detections
+
+  - Bundling script: `x-pack/plugins/security_solution/scripts/openapi/bundle_detections.js`
+  - Bundles location: `x-pack/plugins/security_solution/docs/openapi/{ess|serverless}`
+
+- Security Endpoint Exceptions
+
+  - Bundling script: `packages/kbn-securitysolution-endpoint-exceptions-common/scripts/openapi_bundle.js`
+  - Bundles location: `packages/kbn-securitysolution-endpoint-exceptions-common/docs/openapi/{ess|serverless}`
+
+- Security Endpoint Management
+
+  - Bundling script: `x-pack/plugins/security_solution/scripts/openapi/bundle_endpoint_management.js`
+  - Bundles location: `x-pack/plugins/security_solution/docs/openapi/{ess|serverless}`
+
+- Security Endpoint Management
+
+  - Bundling script: `x-pack/plugins/security_solution/scripts/openapi/bundle_entity_analytics.js`
+  - Bundles location: `x-pack/plugins/security_solution/docs/openapi/{ess|serverless}`
+
+- Security Security Exceptions
+
+  - Bundling script: `packages/kbn-securitysolution-exceptions-common/scripts/openapi_bundle.js`
+  - Bundles location: `packages/kbn-securitysolution-exceptions-common/docs/openapi/{ess|serverless}`
+
+- Security Lists
+
+  - Bundling script: `packages/kbn-securitysolution-lists-common/scripts/openapi_bundle.js`
+  - Bundles location: `packages/kbn-securitysolution-lists-common/docs/openapi/{ess|serverless}`
+
+- Security Osquery
+
+  - Bundling script: `x-pack/plugins/osquery/scripts/openapi/bundle.js`
+  - Bundles location: `x-pack/plugins/osquery/docs/openapi/{ess|serverless}`
+
+- Security Timeline
+
+  - Bundling script: `x-pack/plugins/security_solution/scripts/openapi/bundle_timeline.js`
+  - Bundles location: `x-pack/plugins/security_solution/docs/openapi/{ess|serverless}`