Allow TransportConfigUpdateAction when security config initialization has completed

[cwperks]

Description

Introduces another variable on DynamicConfigFactory called bgThreadComplete that behaves differently than the initialized variable. bgThreadComplete is a flag that signals to TransportConfigUpdateAction that it can start accepting updates.

There are 2 ways the security index can be created from scratch:

1. If plugins.security.allow_default_init_securityindex is set to true it will create the security index and load all yaml files
2. If plugins.security.allow_default_init_securityindex is set to false, the security index is not created on bootstrap and requires a user to run securityadmin to initialize security. When securityadmin is utilized, the cluster does depend on TransportConfigUpdateAction to initialize security so there still needs to be an avenue where this can update the config before initialized is set to true

This PR sets bgThreadComplete to false on node startup and explicitly sets it to true once its ok for TransportConfigUpdateAction to start accepting transport actions. In case 2) up above, it can be set to true before DynamicConfigFactory is initialized so that it can accept requests from securityadmin.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Bug fix

Issues Resolved

• Resolves [BUG] TransportConfigUpdateAction causing a race condition during node bootstrap leading to 403/500 errors #3204

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (1f9edf4) 66.58% compared to head (4d53edc) 65.17%.
Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3810      +/-   ##
==========================================
- Coverage   66.58%   65.17%   -1.42%     
==========================================
  Files         298      298              
  Lines       21188    21218      +30     
  Branches     3453     3460       +7     
==========================================
- Hits        14109    13828     -281     
- Misses       5363     5674     +311     
  Partials     1716     1716              

Files	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	84.48% <100.00%> (+0.04%)	⬆️
...ch/security/securityconf/DynamicConfigFactory.java	55.62% <ø> (ø)
...g/opensearch/security/support/ConfigConstants.java	95.23% <ø> (ø)
...tion/configupdate/TransportConfigUpdateAction.java	95.45% <66.66%> (-4.55%)	⬇️
...earch/security/privileges/PrivilegesEvaluator.java	72.03% <0.00%> (-0.53%)	⬇️
...ecurity/configuration/ConfigurationRepository.java	69.11% <65.55%> (-10.35%)	⬇️

... and 37 files with indirect coverage changes

[cwperks]

Looking at the CI failures

[peternied]

I'm concerned these test cases have will be flaky due to system performance differences on github runners. Lets make sure the validation is really solid since we are fixing a timing issue based problem.

[peternied]

I've created a PR [1] that I think helps clean up the logic of this class so there is a set way that the background initialization is triggered - its always bugged me that we start a thread in the constructor but then 'restart it' after init is called. Let me know what you think of this approach.

• [1] https://github.com/cwperks/security/pull/17/files?diff=unified&w=1

[peternied]

@scrawfor99 Could you take another look?

[stephen-crawford]

Great work Craig!

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-3810-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 045d4ef8a5d9bb8470e03c1b4d7cc68847b986cb
# Push it to GitHub
git push --set-upstream origin backport/backport-3810-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3810-to-2.x.