[Cloud Security] add is_internal config option for outputs #175546

maxcold · 2024-01-25T14:12:07Z

Summary

part of [Cloud Security] Hide internal Fleet server host URL and ES Output #165251

introducing a new is_internal config option for xpack.fleet.outputs. The usage is currently to protect the internal outputs in the UI:

filter out internal outputs in the Settings UI
disable internal outputs in output select for an agent policy

Screencast

screencast-github.com-2024.01.26-15_57_56.webm

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list

…ttings

maxcold · 2024-01-26T10:10:21Z

/ci

…te --fix'

maxcold · 2024-01-26T13:12:42Z

/ci

elasticmachine · 2024-01-26T14:59:59Z

Pinging @elastic/fleet (Team:Fleet)

elasticmachine · 2024-01-26T15:00:41Z

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

maxcold · 2024-01-26T15:01:58Z

packages/kbn-check-mappings-update-cli/current_fields.json

@@ -523,6 +523,7 @@
    "hosts",
    "is_default",
    "is_default_monitoring",
+    "is_internal",


tbh not sure how to test/check this change, just following the pattern for other config keys

You manually modified this file right?

This check was added to make sure that any mapping addition is properly done, by adding a new model version. And I don't see that being done in this PR (looking at x-pack/plugins/fleet/server/saved_objects/index.ts)

Please follow the guidelines available in the model version documentation

(or follow what was already done for this SO type:

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 312 to 321 in 70508b9

'2': {

changes: [

{

type: 'mappings_addition',

addedMappings: {

service_token: { type: 'keyword', index: false },

},

},

],

},

)

thanks for the pointer, will check and follow the guidelines!

@pgayvallet added a new model version to reflect the addition of the is_internal mapping. Btw running node scripts/check_mappings_update --fix made quite a lot of changes to the current_mappings.json which are not related to the is_internal, so I ended up still adding

"is_internal": { "type": "boolean", "index": false },

manually (node scripts/check_mappings_update passes after I added the model version)

kpollich

Code LGTM - one minor suggestion on the docs wording, otherwise 🚀

docs/settings/fleet-settings.asciidoc

Co-authored-by: Kyle Pollich <[email protected]>

kilfoyle

LGTM for the docs change. 👍

pgayvallet

Mappings were added without corresponding model version

pgayvallet · 2024-01-29T13:50:29Z

packages/kbn-check-mappings-update-cli/current_fields.json

@@ -523,6 +523,7 @@
    "hosts",
    "is_default",
    "is_default_monitoring",
+    "is_internal",


You manually modified this file right?

This check was added to make sure that any mapping addition is properly done, by adding a new model version. And I don't see that being done in this PR (looking at x-pack/plugins/fleet/server/saved_objects/index.ts)

Please follow the guidelines available in the model version documentation

(or follow what was already done for this SO type:

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 312 to 321 in 70508b9

'2': {

changes: [

{

type: 'mappings_addition',

addedMappings: {

service_token: { type: 'keyword', index: false },

},

},

],

},

)

nchaulet

This field should probably be added to that function too so existing output are updated https://github.com/nchaulet/kibana/blob/65375dd33f189e501e7f15b6907d613f536664f6/x-pack/plugins/fleet/server/services/preconfiguration/outputs.ts#L293

maxcold · 2024-01-29T17:29:36Z

@nchaulet thanks for the pointer, added is_internal there and a unit test for that

nchaulet

code LGTM 🚀

pgayvallet

LGTM

eyalkraft

Nice!

… src/core/server/integration_tests/ci_checks'

kibana-ci · 2024-01-31T11:35:53Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4ef8151

Failed CI Steps

Jest Tests #3

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.2MB	1.2MB	+76.0B

History

💔 Build #190365 failed 6c043f8
💔 Build #189988 failed 1f21d1b
💔 Build #189830 failed c135ba2
💚 Build #189643 succeeded 806259b
💔 Build #189540 failed aaedec8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* main: use build hash in FTR tests [Security Solution] Fix moderate typo (elastic#175883) [Fleet] Fix conflicting dynamic template mappings for intermediate objects (elastic#175970) [Visualize] Prevent overwriting managed content (elastic#175274) [SLO] Add/edit form mark optional fields (elastic#175807) skip failing test suite (elastic#175984) [data views] Provide method of excluding data tiers when getting field list (elastic#167946) [Dataset quality] State management (elastic#174906) [Cloud Security] add is_internal config option for outputs (elastic#175546)

…tion (#175983) ## Summary - Follow up after #175546 - Part of #165251 introducing a new `is_internal` config option for `xpack.fleet.fleetServerHosts`. The usage is currently to protect the internal fleet server hosts in the UI: - filter them out in the Settings UI - disable internal hosts in the agent policy form ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: David Kilfoyle <[email protected]>

…tion (elastic#175983) ## Summary - Follow up after elastic#175546 - Part of elastic#165251 introducing a new `is_internal` config option for `xpack.fleet.fleetServerHosts`. The usage is currently to protect the internal fleet server hosts in the UI: - filter them out in the Settings UI - disable internal hosts in the agent policy form ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: David Kilfoyle <[email protected]>

…75546) ## Summary - part of elastic#165251 introducing a new `is_internal` config option for `xpack.fleet.outputs`. The usage is currently to protect the internal outputs in the UI: - filter out internal outputs in the Settings UI - disable internal outputs in output select for an agent policy ### Screencast [screencast-github.com-2024.01.26-15_57_56.webm](https://github.com/elastic/kibana/assets/478762/917b4a76-a48f-4bdc-b3d8-5598f86febf8) ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Kyle Pollich <[email protected]>

…tion (elastic#175983) ## Summary - Follow up after elastic#175546 - Part of elastic#165251 introducing a new `is_internal` config option for `xpack.fleet.fleetServerHosts`. The usage is currently to protect the internal fleet server hosts in the UI: - filter them out in the Settings UI - disable internal hosts in the agent policy form ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: David Kilfoyle <[email protected]>

…75546) ## Summary - part of elastic#165251 introducing a new `is_internal` config option for `xpack.fleet.outputs`. The usage is currently to protect the internal outputs in the UI: - filter out internal outputs in the Settings UI - disable internal outputs in output select for an agent policy ### Screencast [screencast-github.com-2024.01.26-15_57_56.webm](https://github.com/elastic/kibana/assets/478762/917b4a76-a48f-4bdc-b3d8-5598f86febf8) ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Kyle Pollich <[email protected]>

…tion (elastic#175983) ## Summary - Follow up after elastic#175546 - Part of elastic#165251 introducing a new `is_internal` config option for `xpack.fleet.fleetServerHosts`. The usage is currently to protect the internal fleet server hosts in the UI: - filter them out in the Settings UI - disable internal hosts in the agent policy form ### Checklist Delete any items that are not applicable to this PR. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: David Kilfoyle <[email protected]>

add is_internal config option for outputs

7692cfe

maxcold requested a review from a team January 25, 2024 14:12

maxcold added 2 commits January 25, 2024 17:49

add e2e test and disable internal outputs in the edit agent policy se…

df81f0a

…ttings

add is_internal flag to kbn-check-mappings-update-cli

aaedec8

kibanamachine and others added 3 commits January 26, 2024 10:33

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

a3c170f

…te --fix'

fix so integration test

53cb801

fix isInternal checkin useOutputOptions hook

8766ecf

maxcold added 2 commits January 26, 2024 14:22

add test for useOutputOptions to validate internal output handling

ea0c40d

add is_internal description for fleet settings docs

9fb0ff5

maxcold marked this pull request as ready for review January 26, 2024 14:59

maxcold requested review from a team as code owners January 26, 2024 14:59

Merge branch 'main' into csp-hide-internal-output-and-fleet-server-host

63984ec

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 26, 2024

maxcold added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting Team:Cloud Security Cloud Security team related labels Jan 26, 2024

maxcold commented Jan 26, 2024

View reviewed changes

kpollich approved these changes Jan 26, 2024

View reviewed changes

docs/settings/fleet-settings.asciidoc Outdated Show resolved Hide resolved

Update docs wording for is_internal flag

806259b

Co-authored-by: Kyle Pollich <[email protected]>

kilfoyle approved these changes Jan 26, 2024

View reviewed changes

Merge branch 'main' into csp-hide-internal-output-and-fleet-server-host

c135ba2

pgayvallet reviewed Jan 29, 2024

View reviewed changes

nchaulet reviewed Jan 29, 2024

View reviewed changes

generated bundeled.json for fleet openapi docs

229731c

maxcold added 2 commits January 29, 2024 18:28

update preconfigured output in case is_internal changed

ef15d35

Merge branch 'main' into csp-hide-internal-output-and-fleet-server-host

327cdda

add new SO model version with is_internal mapping_additions definition

1f21d1b

nchaulet approved these changes Jan 29, 2024

View reviewed changes

pgayvallet approved these changes Jan 30, 2024

View reviewed changes

Merge branch 'main' into csp-hide-internal-output-and-fleet-server-host

6c043f8

eyalkraft approved these changes Jan 31, 2024

View reviewed changes

maxcold and others added 2 commits January 31, 2024 10:39

fix the is_internal.type on the mapping model version to be boolean

cbc4c05

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

4ef8151

… src/core/server/integration_tests/ci_checks'

maxcold merged commit 7b24ddd into main Jan 31, 2024
21 checks passed

maxcold deleted the csp-hide-internal-output-and-fleet-server-host branch January 31, 2024 12:15

kibanamachine added the v8.13.0 label Jan 31, 2024

maxcold mentioned this pull request Feb 1, 2024

[Cloud Security] Add is_internal flag to fleet server hosts configuration #175983

Merged

3 tasks

maxcold mentioned this pull request Feb 6, 2024

[Cloud Security] Hide internal Fleet server host URL and ES Output #165251

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloud Security] add is_internal config option for outputs #175546

[Cloud Security] add is_internal config option for outputs #175546

maxcold commented Jan 25, 2024 •

edited

Loading

maxcold commented Jan 26, 2024

maxcold commented Jan 26, 2024

elasticmachine commented Jan 26, 2024

elasticmachine commented Jan 26, 2024

maxcold Jan 26, 2024

pgayvallet Jan 29, 2024

maxcold Jan 29, 2024

maxcold Jan 29, 2024

kpollich left a comment

kilfoyle left a comment

pgayvallet left a comment

pgayvallet Jan 29, 2024

nchaulet left a comment

maxcold commented Jan 29, 2024

nchaulet left a comment

pgayvallet left a comment

eyalkraft left a comment

kibana-ci commented Jan 31, 2024

	'2': {
	changes: [
	{
	type: 'mappings_addition',
	addedMappings: {
	service_token: { type: 'keyword', index: false },
	},
	},
	],
	},

[Cloud Security] add is_internal config option for outputs #175546

[Cloud Security] add is_internal config option for outputs #175546

Conversation

maxcold commented Jan 25, 2024 • edited Loading

Summary

Screencast

Checklist

maxcold commented Jan 26, 2024

maxcold commented Jan 26, 2024

elasticmachine commented Jan 26, 2024

elasticmachine commented Jan 26, 2024

maxcold Jan 26, 2024

Choose a reason for hiding this comment

pgayvallet Jan 29, 2024

Choose a reason for hiding this comment

maxcold Jan 29, 2024

Choose a reason for hiding this comment

maxcold Jan 29, 2024

Choose a reason for hiding this comment

kpollich left a comment

Choose a reason for hiding this comment

kilfoyle left a comment

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Jan 29, 2024

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

maxcold commented Jan 29, 2024

nchaulet left a comment

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

eyalkraft left a comment

Choose a reason for hiding this comment

kibana-ci commented Jan 31, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

maxcold commented Jan 25, 2024 •

edited

Loading