[Security Solution] Allow users to edit setup field for custom rules

[approksiu]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Related to: https://github.com/elastic/security-team/issues/6951, https://github.com/elastic/enhancements/issues/15982

Background

Users currently cannot edit the Setup guide field for their custom rules in UI. This prevents them from easily adding relevant information about their rules - they either need to import rules with this field setup, or use the other available text edit fields for this purpose.

User Story / Problem Statement(s)

• Expose the Setup guide field in the Rule edit/create page under Advanced settings.

Designs

https://www.figma.com/file/4OzwpyZYgb7haOKQlSZBql/Allow-users-to-add%2Fedit-Setup-guide-field-for-rules-%236951?type=design&node-id=1%3A40&mode=design&t=jwrbC9hVoYCdmd1S-1

Release progress

• Initial implementation is done. A decision is made regarding using a feature flag vs a long-living feature branch for this feature. (PR)
• Feature is covered with automated tests. (PR, PR)
• Feature is fully implemented and considered by the development team as ready to be released.
• Acceptance testing is done and the feature is approved by @approksiu and @ARWNightingale.
• Exploratory testing is done and the feature is approved by @vgomez-el.
• Documentation is written for ESS and Serverless by @joepeeples. Two docs PRs are approved and ready to be merged. (ticket)
• Feature is released in Serverless.

Planned release date in Serverless: the week of April 15, 2024.
Planned release date in ESS: May 21, 2024 (v8.14.0).

[approksiu]

The Setup guide component should not have the "interactive actions", only editing capability.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

Hey @dplumlee @maximpn @nikitaindik, just a comment regarding all the 4 tickets: #173595, #173594, #173593, and #173626

I triaged and added a Release progress checklist to each of them. There are two concerns I've already discussed with most of you today:

• Since we need to introduce some schema changes for each of these tickets, it's not 100% clear if we should use the tried and proven approach of feature toggling to release these enhancements, or we'd have to do it using feature branches, or combine the two approaches. Please think of pros and cons of these approaches and let's discuss on Monday how we're going to proceed.
• We probably don't need to write a new test plan for these enhancements, because there are no existing test plans for rule creation or editing, and because this is not our area of ownership. But I believe we need to add tests anyway. Let's also align on that on Monday and see what kind of tests we'd need to add.

[jpdjere]

cc @approksiu @banderror

Discussed release approach with the @elastic/security-detection-rule-management team and @joepeeples for Docs:

• The schema changes introduced in these four upcoming PRs follow the pattern of: adding each respective field to either the BaseOptionalFields or BaseDefaultableFields and removing them from ResponseFields. This means that, when creating or editing a rule, the API response will stay the same, while the request payload now no longer ignores the new field and uses it to create/update the field.
  • In practice, the only breaking change we can see here is an edge case in which a user is currently using the API to edit a rule and includes in the payload one of the newly customizable fields, trusting that it will be ignored by the endpoint logic. If the feature is released, that will no longer be the case.
• Creating feature toggles for releasing these features would be a high complexity for the edge case that we're trying to prevent: we would need to split up our rule schema and offer the legacy or modified version depending on the toggle.
• A long-lived feature branch which accumulated all changes in all four tickets (making all four fields customisable) would be hard to maintain, as it would take a considerable amount of time until all PRs were merged and ready to review, and merge back to main. The branch would need constant merges/rebases from main.
• Instead, and in agreement with @joepeeples, we will have each PR author coordinate the release of each feature with the Docs team so that the time that the feature change is live with no documentation is minimised or completely eliminated.

[banderror]

we will have each PR author coordinate the release of each feature with the Docs team so that the time that the feature change is live with no documentation is minimised or completely eliminated.

Sounds good to me 👍 Thanks a lot for the update @jpdjere!

[dplumlee]

Deployment links have been added to the PR in the latest build message for acceptance and exploratory testing. cc @vgomez-el @approksiu

[banderror]

Hey all, @dplumlee's PR was approved about 3 weeks ago from the engineering side and since then it has been waiting for other release-related activities. I'd like to check where are we in the release process and if we're able to release this enhancement next week.

@approksiu As far as I understand you acceptance tested this and we got an approval from you?

@ARWNightingale @vgomez-el @joepeeples How's it going on your side?

We're targeting next Monday, April 8, as the date of release to Serverless. On Monday the release pipeline will start and the release to Serverless production will happen in a couple of days after that. This means that we would like to merge this PR by the end of this Friday, April 5, Pacific Time Zone.

@ARWNightingale @vgomez-el Can you please confirm that you're able to complete acceptance and exploratory testing by EOD April 5, or otherwise let me know how much more time you need?

Also @joepeeples can you please confirm that the docs will be ready by Tuesday, April 9?

[vgomez-el]

Apologies for the late response. I didn't notice about the comments on this ticket.
I have performed the exploratory testing over the feature in the cloud environment created on the PR. Even I will need to perform more tests for ESS when the feature is merged, it is looking good for me so far.

[banderror]

@dplumlee @joepeeples /cc @vgomez-el @approksiu @ARWNightingale

The PR that we merged on Monday was merged too late and didn't make it to the release this week. The 7abe4c2b5ac8 commit went live which is an older one:

So the enhancement should go live next week, Monday Apr 15th.

[joepeeples]

Also @joepeeples can you please confirm that the docs will be ready by Tuesday, April 9?

@banderror Putting this at the top of my list for next week (week of April 15) to try to get serverless docs published on, or around, April 16. TBH so far I haven't been able to work on this yet; it's been hard to adjust to documenting features so far out of the usual release cycle, but we'll get there. Also sorry I missed this earlier; I've been pinged on a lot of active PRs and issues recently and it's been very hard to keep up.

[banderror]

@joepeeples Got it, sorry I didn't know about the pressure we were contributing to. Then, I would suggest to re-target our other fields UI work (related integrations, required fields, maybe even max signals) to 8.15.0 to make more room for writing documentation and for testing. I'll raise this topic at the team sync.

[approksiu]

@banderror Setup guide has the highest priority. I am ok with the other fields being released next time.

[joepeeples]

@banderror and @approksiu Thanks for being flexible. It turns out the docs we needed for the UI were pretty minimal, so I was able to open the serverless and classic/ESS PRs today. We should be on track to publish tomorrow (Apr 16) to coincide with the field's release to serverless prod.

API docs update might take a little longer. We can discuss in the next team sync.

[banderror]

The feature went live in Serverless.