ESS support for FTR serverless tests. SSL support in kbn/es. kbn/es D…

…X improvements. (#162673) Closes #162593 Closes #163939 Closes #162625 The original intention of this PR was to add FTR support for ESS. However the scope increased as that also required adding SSL support due to tests failing from disabled `security` and no authentication. Additionally, after using serverless in `kbn/es` extensively for this, there was a bit of friction in regards to DX. ## Summary - Switch `x-pack/test_serverless` FTR to use ES serverless instead of (stateful) snapshot - Adds SSL support to Docker and Serverless in `kbn/es` - Adds `port` option override - Adds `teardown` option to kill running nodes if the process exits without shutdown - Adds `kill` option to kill running nodes on startup if detected - Adds `--esFrom serverless` to FTR CLI - Adds `files` option to mount extra files into containers - For serverless, automatically attach to first node with `docker logs -f es01` on startup for better DX. - Added `background` flag to not attach `logs`. - Adds graceful shutdown for ESS cluster - Separate `docker pull` from `run` for better logging, ensures latest image and stops multiple pulls of the same image occurring in parallel - Align (most) default settings for ES serverless with `gradlew` [settings](https://github.com/elastic/elasticsearch-serverless/blob/main/serverless-build-tools/src/main/kotlin/elasticsearch.serverless-run.gradle.kts#L8) - Fixes Docker bind mount permissions in CI - Fixes issue where `esFrom` would default to `snapshot` and override FTR config settings. ### Checklist - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ## Related Issues for Skipped Tests Security Threat Hunting: #165135 Observability: #165138 Response Ops: #165145 --------- Co-authored-by: Dzmitry Lemechko <[email protected]> Co-authored-by: Tiago Costa <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]>
elastic · Aug 30, 2023 · 06ebc31 · 06ebc31
1 parent 681c2e9
commit 06ebc31
Show file tree

Hide file tree

Showing 85 changed files with 2,411 additions and 606 deletions.
diff --git a/.buildkite/pipelines/pull_request/base.yml b/.buildkite/pipelines/pull_request/base.yml
@@ -110,19 +110,20 @@ steps:
     artifact_paths:
       - "target/kibana-security-solution/**/*"
 
-  - command: .buildkite/scripts/steps/functional/security_serverless_defend_workflows.sh
-    label: 'Serverless Security Defend Workflows Cypress Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 40
-    soft_fail: true
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-    artifact_paths:
-      - "target/kibana-security-solution/**/*"
+  # status_exception: Native role management is not enabled in this Elasticsearch instance
+  # - command: .buildkite/scripts/steps/functional/security_serverless_defend_workflows.sh
+  #   label: 'Serverless Security Defend Workflows Cypress Tests'
+  #   agents:
+  #     queue: n2-4-spot
+  #   depends_on: build
+  #   timeout_in_minutes: 40
+  #   soft_fail: true
+  #   retry:
+  #     automatic:
+  #       - exit_status: '*'
+  #         limit: 1
+  #   artifact_paths:
+  #     - "target/kibana-security-solution/**/*"
 
   - command: .buildkite/scripts/steps/functional/security_serverless_investigations.sh
     label: 'Serverless Security Investigations Cypress Tests'

diff --git a/.buildkite/pipelines/pull_request/defend_workflows.yml b/.buildkite/pipelines/pull_request/defend_workflows.yml
@@ -4,7 +4,7 @@ steps:
     agents:
       queue: n2-4-spot
     depends_on: build
-    timeout_in_minutes: 120
+    timeout_in_minutes: 60
     parallelism: 2
     retry:
       automatic:
@@ -18,7 +18,7 @@ steps:
     agents:
       queue: n2-4-virt
     depends_on: build
-    timeout_in_minutes: 120
+    timeout_in_minutes: 60
     parallelism: 6
     retry:
       automatic:

diff --git a/.buildkite/pipelines/pull_request/osquery_cypress.yml b/.buildkite/pipelines/pull_request/osquery_cypress.yml
@@ -25,16 +25,17 @@ steps:
     artifact_paths:
       - "target/kibana-osquery/**/*"
 
-  - command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh
-    label: 'Serverless Osquery Cypress Tests'
-    agents:
-      queue: n2-4-spot
-    depends_on: build
-    timeout_in_minutes: 50
-    parallelism: 6
-    retry:
-      automatic:
-        - exit_status: '*'
-          limit: 1
-    artifact_paths:
-      - "target/kibana-osquery/**/*"
+  # Error: self-signed certificate in certificate chain
+  # - command: .buildkite/scripts/steps/functional/security_serverless_osquery.sh
+  #   label: 'Serverless Osquery Cypress Tests'
+  #   agents:
+  #     queue: n2-4-spot
+  #   depends_on: build
+  #   timeout_in_minutes: 50
+  #   parallelism: 6
+  #   retry:
+  #     automatic:
+  #       - exit_status: '*'
+  #         limit: 1
+  #   artifact_paths:
+  #     - "target/kibana-osquery/**/*"
diff --git a/.buildkite/scripts/steps/functional/common.sh b/.buildkite/scripts/steps/functional/common.sh
@@ -21,3 +21,7 @@ if [[ -d "$cacheDir" ]]; then
 fi
 
 is_test_execution_step
+
+# logins into docker as a common step for functional tests
+echo "$KIBANA_DOCKER_PASSWORD" | docker login -u "$KIBANA_DOCKER_USERNAME" --password-stdin docker.elastic.co
+trap 'docker logout docker.elastic.co' EXIT
diff --git a/.buildkite/scripts/steps/functional/security_serverless_osquery.sh b/.buildkite/scripts/steps/functional/security_serverless_osquery.sh
@@ -2,10 +2,13 @@
 
 set -euo pipefail
 
-source .buildkite/scripts/common/util.sh
+source .buildkite/scripts/steps/functional/common.sh
 source .buildkite/scripts/steps/functional/common_cypress.sh
 
-.buildkite/scripts/bootstrap.sh
+# TODO: remove the line below to use build artifacts for tests.
+# in addition to remove the line, we will have to expose the kibana install dir into the downloaded build location
+# by exporting a var like:
+# export KIBANA_INSTALL_DIR=${KIBANA_BUILD_LOCATION}
 node scripts/build_kibana_platform_plugins.js
 
 export JOB=kibana-osquery-cypress-serverless

diff --git a/packages/kbn-dev-utils/index.ts b/packages/kbn-dev-utils/index.ts
@@ -19,6 +19,7 @@ export {
   KBN_P12_PATH,
   KBN_P12_PASSWORD,
 } from './src/certs';
+export * from './src/dev_service_account';
 export * from './src/axios';
 export * from './src/plugin_list';
 export * from './src/streams';

diff --git a/packages/kbn-dev-utils/src/dev_service_account.ts b/packages/kbn-dev-utils/src/dev_service_account.ts
@@ -0,0 +1,19 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+const env = process.env;
+
+/**
+ * `kibana-dev` service account token for connecting to ESS
+ * See packages/kbn-es/src/ess_resources/README.md
+ */
+export const kibanaDevServiceAccount = {
+  token:
+    env.TEST_KIBANA_SERVICE_ACCOUNT_TOKEN ||
+    'AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYS1kZXY6VVVVVVVVTEstKiBaNA',
+};
diff --git a/packages/kbn-es/index.ts b/packages/kbn-es/index.ts
@@ -8,4 +8,9 @@
 
 export { run } from './src/cli';
 export { Cluster } from './src/cluster';
-export { SYSTEM_INDICES_SUPERUSER } from './src/utils';
+export {
+  SYSTEM_INDICES_SUPERUSER,
+  ELASTIC_SERVERLESS_SUPERUSER,
+  ELASTIC_SERVERLESS_SUPERUSER_PASSWORD,
+  getDockerFileMountPath,
+} from './src/utils';
diff --git a/packages/kbn-es/src/cli_commands/docker.ts b/packages/kbn-es/src/cli_commands/docker.ts
@@ -12,7 +12,7 @@ import { ToolingLog } from '@kbn/tooling-log';
 import { getTimeReporter } from '@kbn/ci-stats-reporter';
 
 import { Cluster } from '../cluster';
-import { DOCKER_IMG, DOCKER_REPO, DOCKER_TAG } from '../utils';
+import { DOCKER_IMG, DOCKER_REPO, DOCKER_TAG, DEFAULT_PORT } from '../utils';
 import { Command } from './types';
 
 export const docker: Command = {
@@ -27,8 +27,12 @@ export const docker: Command = {
       --tag               Image tag of ES to run from ${DOCKER_REPO} [default: ${DOCKER_TAG}]
       --image             Full path to image of ES to run, has precedence over tag. [default: ${DOCKER_IMG}]
       --password          Sets password for elastic user [default: ${password}]
+      --port              The port to bind to on 127.0.0.1 [default: ${DEFAULT_PORT}]
+      --ssl               Sets up SSL on Elasticsearch
+      --kill              Kill running ES nodes if detected
       -E                  Additional key=value settings to pass to Elasticsearch
       -D                  Override Docker command
+      -F                  Absolute paths for files to mount into container
 
     Examples:
 
@@ -50,9 +54,11 @@ export const docker: Command = {
       alias: {
         esArgs: 'E',
         dockerCmd: 'D',
+        files: 'F',
       },
 
       string: ['tag', 'image', 'D'],
+      boolean: ['ssl', 'kill'],
 
       default: defaults,
     });

diff --git a/packages/kbn-es/src/cli_commands/serverless.ts b/packages/kbn-es/src/cli_commands/serverless.ts
@@ -12,7 +12,7 @@ import { ToolingLog } from '@kbn/tooling-log';
 import { getTimeReporter } from '@kbn/ci-stats-reporter';
 
 import { Cluster } from '../cluster';
-import { SERVERLESS_REPO, SERVERLESS_TAG, SERVERLESS_IMG } from '../utils';
+import { SERVERLESS_REPO, SERVERLESS_TAG, SERVERLESS_IMG, DEFAULT_PORT } from '../utils';
 import { Command } from './types';
 
 export const serverless: Command = {
@@ -22,10 +22,15 @@ export const serverless: Command = {
     return dedent`
     Options:
 
-      --tag               Image tag of ES Serverless to run from ${SERVERLESS_REPO} [default: ${SERVERLESS_TAG}]
-      --image             Full path of ES Serverless image to run, has precedence over tag. [default: ${SERVERLESS_IMG}]
+      --tag               Image tag of ESS to run from ${SERVERLESS_REPO} [default: ${SERVERLESS_TAG}]
+      --image             Full path of ESS image to run, has precedence over tag. [default: ${SERVERLESS_IMG}]
       --clean             Remove existing file system object store before running
+      --port              The port to bind to on 127.0.0.1 [default: ${DEFAULT_PORT}]
+      --ssl               Sets up SSL on Elasticsearch
+      --kill              Kill running ESS nodes if detected
+      --background        Start ESS without attaching to the first node's logs
       -E                  Additional key=value settings to pass to Elasticsearch
+      -F                  Absolute paths for files to mount into containers
 
     Examples:
 
@@ -46,10 +51,11 @@ export const serverless: Command = {
       alias: {
         basePath: 'base-path',
         esArgs: 'E',
+        files: 'F',
       },
 
       string: ['tag', 'image'],
-      boolean: ['clean'],
+      boolean: ['clean', 'ssl', 'kill', 'background'],
 
       default: defaults,
     });

diff --git a/packages/kbn-es/src/cluster.js b/packages/kbn-es/src/cluster.js
@@ -16,13 +16,15 @@ const { Client } = require('@elastic/elasticsearch');
 const { downloadSnapshot, installSnapshot, installSource, installArchive } = require('./install');
 const { ES_BIN, ES_PLUGIN_BIN, ES_KEYSTORE_BIN } = require('./paths');
 const {
-  log: defaultLog,
-  parseEsLog,
   extractConfigFiles,
+  log: defaultLog,
   NativeRealm,
+  parseEsLog,
   parseTimeoutToMs,
-  runServerlessCluster,
   runDockerContainer,
+  runServerlessCluster,
+  stopServerlessCluster,
+  teardownServerlessClusterSync,
 } = require('./utils');
 const { createCliError } = require('./errors');
 const { promisify } = require('util');
@@ -276,6 +278,10 @@ exports.Cluster = class Cluster {
     }
     this._stopCalled = true;
 
+    if (this._serverlessNodes?.length) {
+      return await stopServerlessCluster(this._log, this._serverlessNodes);
+    }
+
     if (!this._process || !this._outcome) {
       throw new Error('ES has not been started');
     }
@@ -295,6 +301,10 @@ exports.Cluster = class Cluster {
 
     this._stopCalled;
 
+    if (this._serverlessNodes?.length) {
+      return await stopServerlessCluster(this._log, this._serverlessNodes);
+    }
+
     if (!this._process || !this._outcome) {
       throw new Error('ES has not been started');
     }
@@ -573,7 +583,15 @@ exports.Cluster = class Cluster {
       throw new Error('ES has already been started');
     }
 
-    await runServerlessCluster(this._log, options);
+    this._serverlessNodes = await runServerlessCluster(this._log, options);
+
+    if (options.teardown) {
+      /**
+       * Ideally would be async and an event like beforeExit or SIGINT,
+       * but those events are not being triggered in FTR child process.
+       */
+      process.on('exit', () => teardownServerlessClusterSync(this._log, options));
+    }
   }
 
   /**

diff --git a/packages/kbn-es/src/ess_resources/README.md b/packages/kbn-es/src/ess_resources/README.md
@@ -0,0 +1,49 @@
+# Elasticsearch Serverless Resources
+The resources in this directory are used for seeding Elasticsearch Serverless (ESS) images with users, roles and tokens for SSL and authentication. ESS requires file realm authentication, so we will bind mount them into the containers at `/usr/share/elasticsearch/config/`.
+
+## Users
+
+### Default user
+
+The default superuser authentication to login to Kibana is:
+
+```
+username: elastic_serverless
+password: changeme
+```
+
+### Adding users
+
+1. Add the user:encrypted_password to `users` file. The encrypted password for `elastic_serverless` is `changeme` if you want to reuse the value.
+1. Set the new user's roles in `users_roles` file.
+1. Add the username to `operator_users.yml` in the array for file realm users.
+
+
+## Service Account and Tokens
+
+This section for Service Accounts was originally from the [ESS repository](https://github.com/elastic/elasticsearch-serverless/blob/main/serverless-build-tools/src/main/resources/README.service_tokens.md).
+
+The "service_tokens" file contains this line:
+```
+elastic/kibana/kibana-dev:$2a$10$mY2RuGROhk56vLNh.Mgwue98BnkdQPlTR.yGh38ao5jhPJobvuBCq
+```
+
+That line defines a single service token
+- For the `elastic/kibana` service account
+- The token is named `kibana-dev`
+- The token's secret is hashed using bcrypt (`$2a$`) using `10` rounds
+
+Although Elasticsearch used PBKDF2_STRETCH by default, the k8s controller
+creates tokens using bcrypt, so we mimic that here.
+
+The hash is not reversible, so this README is here to tell you what the secret is.
+The secret value is: `UUUUUULK-* Z4`
+That produces an encoded token of: `AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYS1kZXY6VVVVVVVVTEstKiBaNA`
+Yes, the secret was specially chosen to produce an encoded value that can be more easily recognised in development.
+
+If a node is configured to use this `service_tokens` file, then you can authenticate to it with
+```
+curl -H "Authorization: Bearer AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYS1kZXY6VVVVVVVVTEstKiBaNA" http://localhost:9200/_security/_authenticate
+```
+
+The name of the token (`kibana-dev`) is important because the `operator_users.yml` file designates that token as an operator and allows us to seed an ESS cluster with this token.
diff --git a/packages/kbn-es/src/ess_resources/jwks.json b/packages/kbn-es/src/ess_resources/jwks.json
@@ -0,0 +1,10 @@
+{
+  "keys": [
+    {
+      "kty": "RSA",
+      "e": "AQAB",
+      "use": "sig",
+      "n": "v9-88aGdE4E85PuEycxTA6LkM3TBvNScoeP6A-dd0Myo6-LfBlp1r7BPBWmvi_SC6Zam3U1LE3AekDMwqJg304my0pvh8wOwlmRpgKXDXjvj4s59vdeVNhCB9doIthUABd310o9lyb55fWc_qQYE2LK9AyEjicJswafguH6txV4IwSl13ieZAxni0Ca4CwdzXO1Oi34XjHF8F5x_0puTaQzHn5bPG4fiIJN-pwie0Ba4VEDPO5ca4lLXWVi1bn8xMDTAULrBAXJwDaDdS05KMbc4sPlyQPhtY1gcYvUbozUPYxSWwA7fZgFzV_h-uy_oXf1EXttOxSgog1z3cJzf6Q"
+    }
+  ]
+}
diff --git a/packages/kbn-es/src/ess_resources/operator_users.yml b/packages/kbn-es/src/ess_resources/operator_users.yml
@@ -0,0 +1,9 @@
+operator:
+  - usernames: ["elastic_serverless", "system_indices_superuser"]
+    realm_type: "file"
+    auth_type: "realm"
+  - usernames: [ "elastic/kibana" ]
+    realm_type: "_service_account"
+    auth_type: "token"
+    token_source: "file"
+    token_names: [ "kibana-dev" ]
diff --git a/packages/kbn-es/src/ess_resources/role_mapping.yml b/packages/kbn-es/src/ess_resources/role_mapping.yml
@@ -0,0 +1,14 @@
+# Role mapping configuration file which has elasticsearch roles as keys
+# that map to one or more user or group distinguished names
+
+#roleA:   this is an elasticsearch role
+#  - groupA-DN  this is a group distinguished name
+#  - groupB-DN
+#  - user1-DN   this is the full user distinguished name
+
+#power_user:
+#  - "cn=admins,dc=example,dc=com"
+#user:
+#  - "cn=users,dc=example,dc=com"
+#  - "cn=admins,dc=example,dc=com"
+#  - "cn=John Doe,cn=other users,dc=example,dc=com"