[8.x] [SecuritySolutions] Create Entity Store 'entities/list' API (#1…

…92806) (#193562)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[SecuritySolutions] Create Entity Store 'entities/list' API
(#192806)](#192806)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pablo
Machado","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-19T12:54:53Z","message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["v9.0.0","release_note:feature","backport:prev-minor","Theme:
entity_analytics","Feature:Entity Analytics","Team:Entity
Analytics","v8.16.0"],"number":192806,"url":"https://github.com/elastic/kibana/pull/192806","mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192806","number":192806,"mergeCommit":{"message":"[SecuritySolutions]
Create Entity Store 'entities/list' API (#192806)\n\nThis PR introduces
the following API routes for listing Entity
Store\r\n\"entities\":\r\n\r\n<meta charset=\"utf-8\"><b
style=\"font-weight:normal;\"\r\nid=\"docs-internal-guid-9410c5d7-7fff-e873-6830-887939a306fb\"><div\r\ndir=\"ltr\"
style=\"margin-left:-0.75pt;\" align=\"left\">\r\nList Entities | GET
/api/entity_store/entities/list\r\n-- | --\r\n</div></b>\r\n\r\nThe PR
includes the following:\r\n - The OpenAPI schemas for the route\r\n -
The actual Kibana side endpoint\r\n - Add searchEntities function to the
`EntityStoreDataClient`\r\n \r\n\r\n### How to test\r\n\r\n1. Add some
host/user data\r\n* Easiest is to
use\r\n[elastic/security-data-generator](https://github.com/elastic/security-documents-generator)\r\n2.
Make sure to add `entityStoreEnabled`
under\r\n`xpack.securitySolution.enableExperimental` in your
`kibana.dev.yml`\r\n3. In kibana dev tools or your terminal, call the
`INIT` route for\r\neither `user` or `host`.\r\n4. You should now see 2
transforms in kibana. Make sure to re-trigger\r\nthem if needed so they
process the documents.\r\n5. Call the new API, and it should return
entities \r\n\r\n\r\n\r\nImplements
https://github.com/elastic/security-team/issues/10517\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"27f5da436b70da1a3743ee99c54d8159918b40de"}},{"branch":"8.x","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <[email protected]>

.buildkite/ftr_security_serverless_configs.yml

@@ -78,6 +78,7 @@ disabled:
   - x-pack/test/security_solution_api_integration/test_suites/genai/knowledge_base/entries/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts
+  - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/exception_lists_items/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/lists_items/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/explore/hosts/trial_license_complete_tier/configs/serverless.config.ts

.buildkite/ftr_security_stateful_configs.yml

@@ -62,6 +62,7 @@ enabled:
   - x-pack/test/security_solution_api_integration/test_suites/detections_response/user_roles/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/ess.config.ts
+  - x-pack/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/exception_lists_items/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/lists_and_exception_lists/lists_items/trial_license_complete_tier/configs/ess.config.ts
   - x-pack/test/security_solution_api_integration/test_suites/explore/hosts/trial_license_complete_tier/configs/ess.config.ts

oas_docs/output/kibana.serverless.staging.yaml

@@ -8321,6 +8321,85 @@ paths:
       summary: Stop the Entity Store engine
       tags:
         - Security Solution Entity Analytics API
+  /api/entity_store/entities/list:
+    get:
+      description: 'List entities records, paging, sorting and filtering as needed.'
+      operationId: ListEntities
+      parameters:
+        - in: query
+          name: sort_field
+          required: false
+          schema:
+            type: string
+        - in: query
+          name: sort_order
+          required: false
+          schema:
+            enum:
+              - asc
+              - desc
+            type: string
+        - in: query
+          name: page
+          required: false
+          schema:
+            minimum: 1
+            type: integer
+        - in: query
+          name: per_page
+          required: false
+          schema:
+            maximum: 10000
+            minimum: 1
+            type: integer
+        - description: An ES query to filter by.
+          in: query
+          name: filterQuery
+          required: false
+          schema:
+            type: string
+        - in: query
+          name: entities_types
+          required: true
+          schema:
+            items:
+              $ref: >-
+                #/components/schemas/Security_Solution_Entity_Analytics_API_EntityType
+            type: array
+      responses:
+        '200':
+          content:
+            application/json; Elastic-Api-Version=2023-10-31:
+              schema:
+                type: object
+                properties:
+                  inspect:
+                    $ref: >-
+                      #/components/schemas/Security_Solution_Entity_Analytics_API_InspectQuery
+                  page:
+                    minimum: 1
+                    type: integer
+                  per_page:
+                    maximum: 1000
+                    minimum: 1
+                    type: integer
+                  records:
+                    items:
+                      $ref: >-
+                        #/components/schemas/Security_Solution_Entity_Analytics_API_Entity
+                    type: array
+                  total:
+                    minimum: 0
+                    type: integer
+                required:
+                  - records
+                  - page
+                  - per_page
+                  - total
+          description: Entities returned successfully
+      summary: List Entity Store Entities
+      tags:
+        - Security Solution Entity Analytics API
   /api/exception_lists:
     delete:
       operationId: DeleteExceptionList
@@ -30804,18 +30883,113 @@ components:
         - started
         - stopped
       type: string
+    Security_Solution_Entity_Analytics_API_Entity:
+      oneOf:
+        - $ref: >-
+            #/components/schemas/Security_Solution_Entity_Analytics_API_UserEntity
+        - $ref: >-
+            #/components/schemas/Security_Solution_Entity_Analytics_API_HostEntity
     Security_Solution_Entity_Analytics_API_EntityType:
       enum:
         - user
         - host
       type: string
+    Security_Solution_Entity_Analytics_API_HostEntity:
+      type: object
+      properties:
+        entity:
+          type: object
+          properties:
+            definitionId:
+              type: string
+            definitionVersion:
+              type: string
+            displayName:
+              type: string
+            firstSeenTimestamp:
+              format: date-time
+              type: string
+            id:
+              type: string
+            identityFields:
+              items:
+                type: string
+              type: array
+            lastSeenTimestamp:
+              format: date-time
+              type: string
+            schemaVersion:
+              type: string
+            type:
+              enum:
+                - node
+              type: string
+          required:
+            - lastSeenTimestamp
+            - schemaVersion
+            - definitionVersion
+            - displayName
+            - identityFields
+            - id
+            - type
+            - firstSeenTimestamp
+            - definitionId
+        host:
+          type: object
+          properties:
+            architecture:
+              items:
+                type: string
+              type: array
+            domain:
+              items:
+                type: string
+              type: array
+            hostname:
+              items:
+                type: string
+              type: array
+            id:
+              items:
+                type: string
+              type: array
+            ip:
+              items:
+                type: string
+              type: array
+            mac:
+              items:
+                type: string
+              type: array
+            name:
+              type: string
+            type:
+              items:
+                type: string
+              type: array
+          required:
+            - name
     Security_Solution_Entity_Analytics_API_IdField:
       enum:
         - host.name
         - user.name
       type: string
     Security_Solution_Entity_Analytics_API_IndexPattern:
       type: string
+    Security_Solution_Entity_Analytics_API_InspectQuery:
+      type: object
+      properties:
+        dsl:
+          items:
+            type: string
+          type: array
+        response:
+          items:
+            type: string
+          type: array
+      required:
+        - dsl
+        - response
     Security_Solution_Entity_Analytics_API_RiskEngineScheduleNowErrorResponse:
       type: object
       properties:
@@ -30843,6 +31017,77 @@ components:
       required:
         - status_code
         - message
+    Security_Solution_Entity_Analytics_API_UserEntity:
+      type: object
+      properties:
+        entity:
+          type: object
+          properties:
+            definitionId:
+              type: string
+            definitionVersion:
+              type: string
+            displayName:
+              type: string
+            firstSeenTimestamp:
+              format: date-time
+              type: string
+            id:
+              type: string
+            identityFields:
+              items:
+                type: string
+              type: array
+            lastSeenTimestamp:
+              format: date-time
+              type: string
+            schemaVersion:
+              type: string
+            type:
+              enum:
+                - node
+              type: string
+          required:
+            - lastSeenTimestamp
+            - schemaVersion
+            - definitionVersion
+            - displayName
+            - identityFields
+            - id
+            - type
+            - firstSeenTimestamp
+            - definitionId
+        user:
+          type: object
+          properties:
+            domain:
+              items:
+                type: string
+              type: array
+            email:
+              items:
+                type: string
+              type: array
+            full_name:
+              items:
+                type: string
+              type: array
+            hash:
+              items:
+                type: string
+              type: array
+            id:
+              items:
+                type: string
+              type: array
+            name:
+              type: string
+            roles:
+              items:
+                type: string
+              type: array
+          required:
+            - name
     Security_Solution_Exceptions_API_CreateExceptionListItemComment:
       type: object
       properties: