okta: allow user configuration of debug_data flattened use #9868

efd6 · 2024-05-15T03:56:37Z

Status: Blocked by elastic/kibana#183496 which will require v8.15.0.

Proposed commit message

See title.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [okta.system] Utilize 'subobjects: false' for debugContext.debugData #9863

Screenshots

efd6 · 2024-05-15T04:00:37Z

This change does not currently pass.

--- Test results for package: okta - START ---
FAILURE DETAILS:
okta/system test-okta-system-no-flattened-events.log:
[0] field "okta.debug_context.debug_data.behaviors.new_country" is undefined
[1] field "okta.debug_context.debug_data.behaviors.new_device" is undefined
[2] field "okta.debug_context.debug_data.behaviors.new_geo_location" is undefined
[3] field "okta.debug_context.debug_data.behaviors.new_ip" is undefined
[4] field "okta.debug_context.debug_data.behaviors.new_state" is undefined
[5] field "okta.debug_context.debug_data.behaviors.velocity" is undefined
[6] field "okta.debug_context.debug_data.behaviors.velocity_behavior" is undefined
[7] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_city" is undefined
[8] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_country" is undefined
[9] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_device" is undefined
[10] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_geo_location" is undefined
[11] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_ip" is undefined
[12] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_state" is undefined
[13] field "okta.debug_context.debug_data.log_only_security_data.behaviors.velocity" is undefined
[14] field "okta.debug_context.debug_data.log_only_security_data.risk.level" is undefined
[15] field "okta.debug_context.debug_data.log_only_security_data.risk.reasons" is undefined
[16] field "okta.debug_context.debug_data.original_principal.alternate_id" is undefined
[17] field "okta.debug_context.debug_data.original_principal.display_name" is undefined
[18] field "okta.debug_context.debug_data.original_principal.id" is undefined
[19] field "okta.debug_context.debug_data.original_principal.type" is undefined
[20] field "okta.debug_context.debug_data.prompting_policy_types" is undefined
[21] field "okta.debug_context.debug_data.risk.level" is undefined
[22] field "okta.debug_context.debug_data.risk.reasons" is undefined
[23] field "okta.debug_context.debug_data.risk_object" is undefined



╭─────────┬─────────────┬───────────┬───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT                                                                      │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ okta    │ system      │ pipeline  │ test-okta-system-events.log               │ PASS                                                                        │  17.328337ms │
│ okta    │ system      │ pipeline  │ test-okta-system-no-flattened-events.log  │ FAIL: test case failed: one or more problems with fields found in documents │  34.252526ms │
│ okta    │ system      │ pipeline  │ test-okta-system-yes-flattened-events.log │ PASS                                                                        │  28.417599ms │
│ okta    │ system      │ pipeline  │ (ingest pipeline warnings)                │ PASS                                                                        │ 395.535137ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯

So add one of these to fix this…

diff --git a/packages/okta/data_stream/system/fields/fields.yml b/packages/okta/data_stream/system/fields/fields.yml
index b0f51b1b0..c314f671c 100644
--- a/packages/okta/data_stream/system/fields/fields.yml
+++ b/packages/okta/data_stream/system/fields/fields.yml
@@ -273,3 +273,5 @@
   fields:
     - name: ip_chain
       type: flattened
+- name: okta.debug_context.debug_data.behaviors.new_city
+  type: keyword
\ No newline at end of file
diff --git a/packages/okta/docs/README.md b/packages/okta/docs/README.md
index 00f10e7ea..9762a1b64 100644
--- a/packages/okta/docs/README.md
+++ b/packages/okta/docs/README.md
@@ -304,6 +304,7 @@ An example event for `system` looks as following:
 | okta.client.zone | The zone information of the client. | keyword |
 | okta.debug_context.debug_data |  | object |
 | okta.debug_context.debug_data.authn_request_id | The authorization request ID. | keyword |
+| okta.debug_context.debug_data.behaviors.new_city |  | keyword |
 | okta.debug_context.debug_data.device_fingerprint | The fingerprint of the device. | keyword |
 | okta.debug_context.debug_data.dt_hash | The device token hash | keyword |
 | okta.debug_context.debug_data.factor | The factor used for authentication. | keyword |

This however causes a mapping failure:

Error: error running package system tests: could not complete test run: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"mapper_parsing_exception\n\tCaused by:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects\n\tRoot causes:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects"}

mapper_parsing_exception
	Caused by:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects
	Root causes:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects

A minimal version of this is here:

- name: grandparent
  type: object
  object_type: keyword
  object_type_mapping_type: "*"
  subobjects: false
- name: grandparent.parent.child
  type: keyword

elasticmachine · 2024-05-15T04:51:47Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-05-15T04:59:16Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/okta/data_stream/system/fields/fields.yml

packages/okta/data_stream/system/elasticsearch/ingest_pipeline/no_use_flattened_debug.yml

packages/okta/data_stream/system/manifest.yml

andrewkroh · 2024-05-21T00:22:07Z

I was testing the upgrade process going from 2.9.0 to 2.10.0 (751ce10) where I already had some data indexed, and the upgrade (using 8.13.4) fails with

mapper_exception: the [subobjects] parameter can't be updated for the object mapping [okta.debug_context.debug_data]

@zmoog @ruflin Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

andrewkroh · 2024-05-21T04:53:21Z

packages/okta/manifest.yml

@@ -138,6 +138,13 @@ policy_templates:
            multi: false
            required: false
            show_user: true
+          - name: remove_flattened_debug


remove_flattened_debug is declared in two places. I think one of them needs to be removed.

Relates: elastic/package-spec#421

elastic-sonarqube · 2024-05-21T06:17:42Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

zmoog · 2024-05-21T06:25:56Z

Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

Yes, Fleet should do a rollover.

@flash1293 mentioned this issue during our sync, a couple of days ago. We are looking into this.

flash1293 · 2024-05-21T06:56:24Z

Sorry for this hurdle, the fleet team is already working on it here: elastic/kibana#183496

andrewkroh · 2024-06-23T14:39:48Z

@flash1293 Now that 8.14 is out, could we avoid the problem of elastic/kibana#183496 if we switched this data stream over to using data subobjects: false at the data stream level instead of doing it at the per-field level? Not sure if that changes anything, but wanted to see if it would unblock this.

elastic-sonarqube · 2024-06-23T21:21:37Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

flash1293 · 2024-06-24T08:47:56Z

@andrewkroh I don't think this changes anything - in fact the custom integration case is also using subobjects:false on the datastream level.

botelastic · 2024-08-08T03:11:53Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

andrewkroh · 2024-08-09T19:02:20Z

8.15.0 is out now so this should be unblocked by the earlier bug elastic/kibana#183496.

Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. go run github.com/andrewkroh/go-examples/[email protected] -ecs-version=8.11.0 [email protected] -drop-import-mappings -kibana-version=^8.13.0 -fields-yml-drop-ecs packages/okta

elasticmachine · 2024-08-11T21:38:07Z

💚 Build Succeeded

Buildkite Build
Commit: 9fb6beb

History

💚 Build #12738 succeeded c78ad0144e3a2f4bc2786c3afea7bbeddf1af5c6
💚 Build #11710 succeeded 4604582c7c24f1683a286b1e081ff013ad4bf081
💚 Build #11621 succeeded 751ce10474faffe859dc0e2ec5dadc1a819ed266
💔 Build #11620 failed 720cfe86c15ec656bb11f340ba57980a110cdea6
💔 Build #11613 failed 7c922b5e2e664a188664638fd481d26742d474d8
💚 Build #11454 succeeded 283d7b38c9b73d87d480ade7452c84db38281dea

cc @efd6

elastic-sonarqube · 2024-08-11T21:38:10Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
80.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-08-29T22:13:58Z

Package okta - 2.12.0 containing this change is available at https://epr.elastic.co/search?package=okta

efd6 added enhancement New feature or request Integration:okta Okta Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels May 15, 2024

efd6 self-assigned this May 15, 2024

efd6 force-pushed the 9863-okta branch from 88f8137 to ee6e3f0 Compare May 15, 2024 03:57

efd6 marked this pull request as ready for review May 15, 2024 04:59

efd6 requested a review from a team as a code owner May 15, 2024 04:59

andrewkroh reviewed May 17, 2024

View reviewed changes

packages/okta/data_stream/system/manifest.yml Outdated Show resolved Hide resolved

packages/okta/data_stream/system/manifest.yml Outdated Show resolved Hide resolved

efd6 force-pushed the 9863-okta branch from 7c922b5 to 720cfe8 Compare May 17, 2024 04:54

andrewkroh reviewed May 21, 2024

View reviewed changes

efd6 requested a review from andrewkroh May 21, 2024 05:54

narph mentioned this pull request Jun 20, 2024

[SentinelOne] Mapping Activity Data field #10175

Open

efd6 force-pushed the 9863-okta branch from 4604582 to c78ad01 Compare June 23, 2024 21:02

botelastic bot added the Stalled label Aug 8, 2024

efd6 removed the Stalled label Aug 8, 2024

efd6 added 2 commits August 12, 2024 06:45

okta: allow user configuration of debug_data flattened use

fff9b15

okta: clean up field definitions

b9f31ce

efd6 added 6 commits August 12, 2024 06:45

okta: work around mapping

ce4e19a

address pr comments

cd0dbe0

fix space-containing keys

706293f

address pr comment

32b0b64

bump kibana.version

9fb6beb

efd6 force-pushed the 9863-okta branch from c78ad01 to 9fb6beb Compare August 11, 2024 21:16

andrewkroh approved these changes Aug 29, 2024

View reviewed changes

efd6 merged commit da30bc6 into elastic:main Aug 29, 2024
5 checks passed

andrewkroh mentioned this pull request Sep 4, 2024

[Okta]: Update 2.11.0 - > 2.12.0 fails #10972

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

okta: allow user configuration of debug_data flattened use #9868

okta: allow user configuration of debug_data flattened use #9868

efd6 commented May 15, 2024 •

edited by andrewkroh

Loading

efd6 commented May 15, 2024 •

edited

Loading

elasticmachine commented May 15, 2024

elasticmachine commented May 15, 2024

andrewkroh commented May 21, 2024 •

edited

Loading

andrewkroh May 21, 2024

elastic-sonarqube bot commented May 21, 2024

zmoog commented May 21, 2024

flash1293 commented May 21, 2024

andrewkroh commented Jun 23, 2024

elastic-sonarqube bot commented Jun 23, 2024

flash1293 commented Jun 24, 2024

botelastic bot commented Aug 8, 2024

andrewkroh commented Aug 9, 2024

elasticmachine commented Aug 11, 2024

elastic-sonarqube bot commented Aug 11, 2024

elasticmachine commented Aug 29, 2024

okta: allow user configuration of debug_data flattened use #9868

okta: allow user configuration of debug_data flattened use #9868

Conversation

efd6 commented May 15, 2024 • edited by andrewkroh Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

efd6 commented May 15, 2024 • edited Loading

elasticmachine commented May 15, 2024

🚀 Benchmarks report

elasticmachine commented May 15, 2024

andrewkroh commented May 21, 2024 • edited Loading

andrewkroh May 21, 2024

Choose a reason for hiding this comment

elastic-sonarqube bot commented May 21, 2024

Quality Gate passed

zmoog commented May 21, 2024

flash1293 commented May 21, 2024

andrewkroh commented Jun 23, 2024

elastic-sonarqube bot commented Jun 23, 2024

Quality Gate passed

flash1293 commented Jun 24, 2024

botelastic bot commented Aug 8, 2024

andrewkroh commented Aug 9, 2024

elasticmachine commented Aug 11, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Aug 11, 2024

Quality Gate passed

elasticmachine commented Aug 29, 2024

efd6 commented May 15, 2024 •

edited by andrewkroh

Loading

efd6 commented May 15, 2024 •

edited

Loading

andrewkroh commented May 21, 2024 •

edited

Loading