[Cloud Security] CSPM AWS agentless selector #171671

maxcold · 2023-11-21T17:12:07Z

Summary

A part of

https://github.com/elastic/security-team/issues/7969

This PR doesn't cover the fact that with Agentless only Manual - Direct Access Key and Manual - Temprary access key are supported. Hiding other setup options, like Cloudformation, Manual - Assume role or Manual - Shared Credentials`, will be resolved in the next PR. As this feature is hidden behind a feature flag and based on the fact that Agentless is available only for a handful of internal Cloud org ids, it should be safe to merge this PR if code-wise it's ok.

How to test

Make sure to have the FF in your serverless.security.dev.yml (it's basically the kibana.dev.yml but specifically for Serverless Security Projects) enabled. Also specify some serverless project id, to enable the logic of isServerlessEnabled

xpack.fleet.enableExperimental: ['agentless']
xpack.cloud.serverless.project_id: 'some_fake_project_id'

Follow the steps from this comment https://github.com/elastic/security-team/issues/7972#issuecomment-1808096052 to have the Agentless artifacts (agent, policy, output, and fleet server host) locally

After that, you should be able to test the flow. Mind that with agentless only Manual - Direct Access Key and Manual - Temprary access key setup methods are available. Hiding other options will be addressed in the follow ups

Screencast

screencast-maxcold-agentless-selector-pr-b5b706.kb.eu-west-1.aws.qa.elastic.cloud-2023.11.27-10_30_57.webm

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2023-11-27T09:21:10Z

Pinging @elastic/fleet (Team:Fleet)

opauloh · 2023-11-29T21:05:20Z

Currently, we don't support Cloud provider auto-deployment (CloudFormation, Google Cloud Shell, Azure ARM) on the same agent policy.

As the example below, once we save multiple CSPM integrations on the same agent policy, including a CloudFormation integration, the Agent enrollment flyout only shows the "Launch CloudFormation" option and always points to the first CloudFormation installed.

Recordings

Saving manual CSPM AWS - ok

Screen.Recording.2023-11-29.at.12.04.04.PM.mov

Saving a Cloudformation CSPM AWS - ok

Screen.Recording.2023-11-29.at.12.03.01.PM.mov

Saving another manual CSPM AWS - it shows only cloud formation deployment

Screen.Recording.2023-11-29.at.12.02.01.PM.mov

Then it only shows cloud formation deployment for all the CSPM AWS integrations

Screen.Recording.2023-11-29.at.12.05.27.PM.mov

@maxcold - From what I could see on the agent policy below, it looks like the agentless policy agent would recognize all the 2 manual cspm integrations and deploy them individually once the integration is saved. Then the user needs to deploy the CloudFormation integration manually in their own AWS account. If that's the case, I don't see benefits in providing agentless integrations with CloudFormation since it doesn't change anything from the agent-based, shouldn't we provide agentless integration for manual only?

Screen.Recording.2023-11-29.at.12.30.21.PM.mov

Agent Policy:

id: agentless
revision: 5
outputs:
  cloudbeat-es-internal-output:
    type: elasticsearch
    hosts:
      - 'https://es01:9200'
    ssl: {}
fleet:
  hosts:
    - 'https://host.docker.internal:8220'
output_permissions:
  cloudbeat-es-internal-output:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    1d8b9b5c-5043-4913-a27f-5237a8d72c84:
      indices:
        - names:
            - logs-cloud_security_posture.findings-default
          privileges:
            - auto_configure
            - create_doc
    b24d71ed-3b1c-4b71-acea-1ef9d5212bb4:
      indices:
        - names:
            - logs-cloud_security_posture.findings-default
          privileges:
            - auto_configure
            - create_doc
    49169e78-e520-4c7e-be0a-9b831bee66bd:
      indices:
        - names:
            - logs-cloud_security_posture.findings-default
          privileges:
            - auto_configure
            - create_doc
    fc6c36ab-da89-40a2-8814-f6b5033d3e6f:
      indices:
        - names:
            - logs-cloud_security_posture.findings-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: false
    logs: false
    metrics: false
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: j5QcqnIAN6oeIPeyk0qDwQ0CSmJ4J7SOvAx3VVujg8A=
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYq1LTRQHCx/XkQ3QfFDdJgRVGjKLOOM6mYM6LnTxzxFcBAn+6qZ/lcImOS2falUyBYoqyIj/PbdVtrzO9JMilQ==
inputs:
  - id: cloudbeat/cis_aws-cspm-1d8b9b5c-5043-4913-a27f-5237a8d72c84
    name: cspm-manual-1
    revision: 1
    type: cloudbeat/cis_aws
    use_output: cloudbeat-es-internal-output
    meta:
      package:
        name: cloud_security_posture
        version: 1.6.5
    data_stream:
      namespace: default
    package_policy_id: 1d8b9b5c-5043-4913-a27f-5237a8d72c84
    streams:
      - id: >-
          cloudbeat/cis_aws-cloud_security_posture.findings-1d8b9b5c-5043-4913-a27f-5237a8d72c84
        data_stream:
          dataset: cloud_security_posture.findings
          type: logs
        period: 24h
        fetchers:
          - name: aws-iam
          - name: aws-ec2-network
          - name: aws-s3
          - name: aws-trail
          - name: aws-monitoring
          - name: aws-rds
        config:
          v1:
            type: cspm
            deployment: aws
            benchmark: cis_aws
            aws:
              account_type: organization-account
              credentials:
                type: assume_role
                role_arn: test
  - id: cloudbeat/cis_aws-cspm-b24d71ed-3b1c-4b71-acea-1ef9d5212bb4
    name: cspm-cloudformation-single
    revision: 1
    type: cloudbeat/cis_aws
    use_output: cloudbeat-es-internal-output
    meta:
      package:
        name: cloud_security_posture
        version: 1.6.5
    data_stream:
      namespace: default
    package_policy_id: b24d71ed-3b1c-4b71-acea-1ef9d5212bb4
    streams:
      - id: >-
          cloudbeat/cis_aws-cloud_security_posture.findings-b24d71ed-3b1c-4b71-acea-1ef9d5212bb4
        data_stream:
          dataset: cloud_security_posture.findings
          type: logs
        period: 24h
        fetchers:
          - name: aws-iam
          - name: aws-ec2-network
          - name: aws-s3
          - name: aws-trail
          - name: aws-monitoring
          - name: aws-rds
        config:
          v1:
            type: cspm
            deployment: aws
            benchmark: cis_aws
            aws:
              account_type: single-account
              credentials:
                type: cloud_formation
    cloud_formation_template_url: >-
      https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cspm-ACCOUNT_TYPE-8.11.0.yml&stackName=Elastic-Cloud-Security-Posture-Management&param_EnrollmentToken=FLEET_ENROLLMENT_TOKEN&param_FleetUrl=FLEET_URL&param_ElasticAgentVersion=KIBANA_VERSION&param_ElasticArtifactServer=https://artifacts.elastic.co/downloads/beats/elastic-agent
  - id: cloudbeat/cis_aws-cspm-49169e78-e520-4c7e-be0a-9b831bee66bd
    name: cspm-cloudformation-1
    revision: 1
    type: cloudbeat/cis_aws
    use_output: cloudbeat-es-internal-output
    meta:
      package:
        name: cloud_security_posture
        version: 1.6.5
    data_stream:
      namespace: default
    package_policy_id: 49169e78-e520-4c7e-be0a-9b831bee66bd
    streams:
      - id: >-
          cloudbeat/cis_aws-cloud_security_posture.findings-49169e78-e520-4c7e-be0a-9b831bee66bd
        data_stream:
          dataset: cloud_security_posture.findings
          type: logs
        period: 24h
        fetchers:
          - name: aws-iam
          - name: aws-ec2-network
          - name: aws-s3
          - name: aws-trail
          - name: aws-monitoring
          - name: aws-rds
        config:
          v1:
            type: cspm
            deployment: aws
            benchmark: cis_aws
            aws:
              account_type: organization-account
              credentials:
                type: cloud_formation
    cloud_formation_template_url: >-
      https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-cspm-ACCOUNT_TYPE-8.11.0.yml&stackName=Elastic-Cloud-Security-Posture-Management&param_EnrollmentToken=FLEET_ENROLLMENT_TOKEN&param_FleetUrl=FLEET_URL&param_ElasticAgentVersion=KIBANA_VERSION&param_ElasticArtifactServer=https://artifacts.elastic.co/downloads/beats/elastic-agent
  - id: cloudbeat/cis_aws-cspm-fc6c36ab-da89-40a2-8814-f6b5033d3e6f
    name: cspm-manual-2
    revision: 1
    type: cloudbeat/cis_aws
    use_output: cloudbeat-es-internal-output
    meta:
      package:
        name: cloud_security_posture
        version: 1.6.5
    data_stream:
      namespace: default
    package_policy_id: fc6c36ab-da89-40a2-8814-f6b5033d3e6f
    streams:
      - id: >-
          cloudbeat/cis_aws-cloud_security_posture.findings-fc6c36ab-da89-40a2-8814-f6b5033d3e6f
        data_stream:
          dataset: cloud_security_posture.findings
          type: logs
        period: 24h
        fetchers:
          - name: aws-iam
          - name: aws-ec2-network
          - name: aws-s3
          - name: aws-trail
          - name: aws-monitoring
          - name: aws-rds
        config:
          v1:
            type: cspm
            deployment: aws
            benchmark: cis_aws
            aws:
              account_type: organization-account
              credentials:
                type: assume_role
                role_arn: test 2
signed:
  data: >-
    xxx==
  signature: >-
    xxx==
secret_references: []

opauloh

When selecting CloudFormation, we also instruct users to select "New hosts" in the "where to add this integration" section, but that section is hidden on agentless and that might be confusing, should we move the agentless option only for manual?

maxcold · 2023-11-30T09:18:53Z

@opauloh thanks for the thorough review, my bad for not providing enough context. With agentless we only support the Manual > Direct Access Key or Temporary Access Key methods. As the feature is currently hidden via a feature flag, which is not enabled anywhere + agentless is enabled for only a handful of org ids from our team, my plan was to limit the installation option in the next PR to not bloat this one for easier review, I should've stated it in the description. Wdyt, is it acceptable to move forward with this PR and limit the installation options in the follow-up?

opauloh · 2023-11-30T14:31:55Z

As the feature is currently hidden via a feature flag, which is not enabled anywhere + agentless is enabled for only a handful of org ids from our team, my plan was to limit the installation option in the next PR to not bloat this one for easier review, I should've stated it in the description. Wdyt, is it acceptable to move forward with this PR and limit the installation options in the follow-up

Yes, since the feature is behind a Feature Flag I see no problem with addressing that on a follow-up PR

opauloh

code changes LGTM, tested locally

kpollich

Fleet codeowner 🚀

Thanks for all the tests and the screencast - makes it very easy to approve. Code looks great.

…-fix'

maxcold · 2023-12-06T14:53:21Z

Will wait till 8.12 is branched out to avoid having changes in the 8.12 codebase. The change is only relevant only for serverless, and to avoid any risk of breaking the release, will merge it into 8.13 main

kibana-ci · 2023-12-06T15:34:08Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b65d7a2
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-171671-b65d7a274ba3
Security Deployment

Failed CI Steps

Defend Workflows Cypress Tests #13

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`cloudSecurityPosture`	429	431	+2
`fleet`	951	954	+3
total			+5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1095	1096	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`cloudSecurityPosture`	443.8KB	446.8KB	+3.0KB
`fleet`	1.2MB	1.2MB	+922.0B
total			+3.9KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5666	+5666
`total size`	-	5.9MB	+5.9MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	156.0KB	156.3KB	+255.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1213	1214	+1

History

💚 Build #179991 succeeded dc55d19
💚 Build #179929 succeeded 8b26fbd
💚 Build #179103 succeeded 8975fd0
💛 Build #179011 was flaky 50bb376
💚 Build #178901 succeeded 422f360

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Summary Follow up after - #171671 Closes - elastic/security-team#7969 Includes: - limiting setup options for agentless to only Direct Access Keys and Temporary Keys - covering Agentless for edit flow ### How to test Make sure to have the FF in your `serverless.security.dev.yml` (it's similar to the `kibana.dev.yml` but specifically for Serverless Security Projects) enabled. Also specify some serverless project id, to enable the logic of `isServerlessEnabled` ``` xpack.fleet.enableExperimental: ['agentless'] xpack.cloud.serverless.project_id: 'some_fake_project_id' ``` The follow the steps from this comment elastic/security-team#7972 (comment) to have the Agentless artifacts (agent, policy, output, and fleet server host) locally After that, you should be able to test the flow. ### Screencast [screencast-mail.google.com-2023.12.08-16_37_35.webm](https://github.com/elastic/kibana/assets/478762/b94b685f-ed37-4e45-9907-bbd95cb8975a) ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: kibanamachine <[email protected]>

…t policy (#173553) ## Summary Right now every Security project in serverless is created with [an Agentless policy](https://github.com/elastic/project-controller/blob/main/internal/project/security/security_kibana_config.go#L80) (gated be the feature flag based on or org id) and in this policy `is_managed` set to `false`. We in Cloud Security want to make the policy managed. This change is to allow us to do that and still be able to install integrations on the policy. In a nutshell, the logic is to force install integration if the agent policy id is `agentless`. If we are not missing something, it should be safe, as when managed, the agentless agent policy won't be available in the list of Existing Hosts and the only way to pick it for installation in the UI is to implement the same logic we implemented in CSP integration in - #171671 - #172562 Part of: - elastic/security-team#8117 ## Screencast https://github.com/elastic/security-team/assets/478762/c41f2f33-0c43-467f-a54a-8710b26a0abc ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

maxcold added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Nov 21, 2023

maxcold force-pushed the csp-agentless-selector branch from ce9e2a1 to b7327f8 Compare November 22, 2023 13:22

maxcold added ci:project-deploy-security Create a Security Serverless Project ci:build-serverless-image labels Nov 24, 2023

maxcold marked this pull request as ready for review November 27, 2023 09:20

maxcold requested review from a team as code owners November 27, 2023 09:20

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Nov 27, 2023

opauloh requested changes Nov 29, 2023

View reviewed changes

opauloh approved these changes Nov 30, 2023

View reviewed changes

maxcold mentioned this pull request Dec 5, 2023

[Cloud Security] limit setup options for Agentless CSPM #172562

Merged

9 tasks

maxcold added the v8.13.0 label Dec 5, 2023

kpollich approved these changes Dec 6, 2023

View reviewed changes

maxcold and others added 12 commits December 6, 2023 15:45

agentless/agent-based selector

5b11f23

i18n for the SetupType selector

644a7b0

use enum in the check to add hosts step

1d61bde

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

aeb4940

…-fix'

move setup type logic in a hook

7cf8584

rename 'setup type' to 'setup technology'

5e89cc8

move selector component to a separate file

3b642fd

move setup technology logic to a custom hook

201211e

fix imports from fleet-plugin

fcefddc

memoize isAgentlessAvailable

3fbf9f6

move useSetupTechnology hook out

8278170

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

4ac7a24

…-fix'

maxcold added 5 commits December 6, 2023 15:46

handle case of coming from a specific agent policy

b9bd508

add tests for useSetupTechnology hook

4247385

add unit tests to useSetupTechnology in csp

756b257

update agentless policy id

add2b37

fetch agentless policy only in serverless

48b5d37

maxcold force-pushed the csp-agentless-selector branch from dc55d19 to 48b5d37 Compare December 6, 2023 14:49

Merge branch 'main' into csp-agentless-selector

b65d7a2

maxcold merged commit df80372 into main Dec 7, 2023

maxcold deleted the csp-agentless-selector branch December 7, 2023 09:07

maxcold mentioned this pull request Dec 20, 2023

[Cloud Security] Allow force install package policy to agentless agent policy #173553

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloud Security] CSPM AWS agentless selector #171671

[Cloud Security] CSPM AWS agentless selector #171671

maxcold commented Nov 21, 2023 •

edited

Loading

elasticmachine commented Nov 27, 2023

opauloh commented Nov 29, 2023 •

edited

Loading

opauloh left a comment

maxcold commented Nov 30, 2023

opauloh commented Nov 30, 2023

opauloh left a comment

kpollich left a comment

maxcold commented Dec 6, 2023

kibana-ci commented Dec 6, 2023 •

edited

Loading

API count

[Cloud Security] CSPM AWS agentless selector #171671

[Cloud Security] CSPM AWS agentless selector #171671

Conversation

maxcold commented Nov 21, 2023 • edited Loading

Summary

How to test

Screencast

Checklist

elasticmachine commented Nov 27, 2023

opauloh commented Nov 29, 2023 • edited Loading

opauloh left a comment

Choose a reason for hiding this comment

maxcold commented Nov 30, 2023

opauloh commented Nov 30, 2023

opauloh left a comment

Choose a reason for hiding this comment

kpollich left a comment

Choose a reason for hiding this comment

maxcold commented Dec 6, 2023

kibana-ci commented Dec 6, 2023 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Canvas Sharable Runtime

Page load bundle

API count

History

maxcold commented Nov 21, 2023 •

edited

Loading

opauloh commented Nov 29, 2023 •

edited

Loading

kibana-ci commented Dec 6, 2023 •

edited

Loading