Added support for feature flags in opensearch.yml (#4959) #5707

ntantri · 2023-01-05T07:51:25Z

(cherry picked from commit 241bd42)

Description

This change introduces a static store "settings"
in FeatureFlags.java file to enable isEnabled method to fetch flag settings defined in opensearch.yml.

Issues Resolved

Addresses this issue Feature flags should be defined in opensearch.yml #4102

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-05T08:08:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8840/
CommitID: f1aee56
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-05T08:11:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/8841/
CommitID: a6d94fe
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-05T09:21:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/8849/
CommitID: b296a89

dreamer-89 · 2023-01-05T17:17:11Z

CHANGELOG.md

- Add feature flag for extensions ([#5211](https://github.com/opensearch-project/OpenSearch/pull/5211))
 ### Added
- Apply reproducible builds configuration for OpenSearch plugins through gradle plugin ([#4746](https://github.com/opensearch-project/OpenSearch/pull/4746))
- Prevent deletion of snapshots that are backing searchable snapshot indexes ([#5069](https://github.com/opensearch-project/OpenSearch/pull/5069))
- Update to Gradle 7.6 ([#5382](https://github.com/opensearch-project/OpenSearch/pull/5382))
- Reject bulk requests with invalid actions ([#5299](https://github.com/opensearch-project/OpenSearch/issues/5299))
- Add max_shard_size parameter for shrink API ([#5229](https://github.com/opensearch-project/OpenSearch/pull/5229))
- Added jackson dependency to server ([#5366] (https://github.com/opensearch-project/OpenSearch/pull/5366))
- Adding support to register settings dynamically ([#5495](https://github.com/opensearch-project/OpenSearch/pull/5495))
- Adding auto release workflow ([#5582](https://github.com/opensearch-project/OpenSearch/pull/5582))
- Added experimental support for extensions ([#5347](https://github.com/opensearch-project/OpenSearch/pull/5347)), ([#5518](https://github.com/opensearch-project/OpenSearch/pull/5518)), ([#5597](https://github.com/opensearch-project/OpenSearch/pull/5597)), ([#5615](https://github.com/opensearch-project/OpenSearch/pull/5615)))
- Add CI bundle pattern to distribution download ([#5348](https://github.com/opensearch-project/OpenSearch/pull/5348))
- Added @gbbafna as an OpenSearch maintainer ([#5668](https://github.com/opensearch-project/OpenSearch/pull/5668))
+- Add feature flag for extensions ([#5211](https://github.com/opensearch-project/OpenSearch/pull/5211))
+- Add support for s390x architecture ([#4001](https://github.com/opensearch-project/OpenSearch/pull/4001))
+- Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
+- Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
+- Add failover support with Segment Replication enabled. ([#4325](https://github.com/opensearch-project/OpenSearch/pull/4325)
+- Point in time rest layer changes for list PIT and PIT segments API ([#4388](https://github.com/opensearch-project/OpenSearch/pull/4388))
+- Added @dreamer-89 as an Opensearch maintainer ([#4342](https://github.com/opensearch-project/OpenSearch/pull/4342))
+- Added release notes for 1.3.5 ([#4343](https://github.com/opensearch-project/OpenSearch/pull/4343))
+- Added release notes for 2.2.1 ([#4344](https://github.com/opensearch-project/OpenSearch/pull/4344))
+- Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
+- Support for HTTP/2 (server-side) ([#3847](https://github.com/opensearch-project/OpenSearch/pull/3847))
+- BWC version 2.2.2 ([#4383](https://github.com/opensearch-project/OpenSearch/pull/4383))
+- Support for labels on version bump PRs, skip label support for changelog verifier ([#4391](https://github.com/opensearch-project/OpenSearch/pull/4391))
+- Add a new node role 'search' which is dedicated to provide search capability ([#4689](https://github.com/opensearch-project/OpenSearch/pull/4689))
+- Introduce experimental searchable snapshot API ([#4680](https://github.com/opensearch-project/OpenSearch/pull/4680))
+- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))


@tan31989 : Looks like lot many incorrect (entries from main?) are also been backported. Can you please fix this ?

@dreamer-89 I verified these manually; only two didn't have the file changes in the 2.x. I have removed those 2 from the CHANGELOG.md. The rest of these have files in the 2.x branch but are never mentioned in the CHANGELOG.md.

I have just "cherry-picked" by committing via command for backport.

Does this mean, I need to keep it here or remove those?

Just FYI, now trying to fix the merge conflicts and failing tests once I had to rebase this branch to 2.x, hence there are multiple pushes to this PR.

Thanks @tan31989 for the effort on this PR.

@dreamer-89 I verified these manually; only two didn't have the file changes in the 2.x. I have removed those 2 from the CHANGELOG.md. The rest of these have files in the 2.x branch but are never mentioned in the CHANGELOG.md.

I see. Thanks for fixing the missing changelog entries.

github-actions · 2023-01-08T05:11:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9011/
CommitID: 2e93b82

codecov-commenter · 2023-01-08T05:12:01Z

Codecov Report

Merging #5707 (2e93b82) into 2.x (ee33c9e) will decrease coverage by 0.51%.
The diff coverage is 66.66%.

@@             Coverage Diff              @@
##                2.x    #5707      +/-   ##
============================================
- Coverage     71.02%   70.50%   -0.52%     
+ Complexity    58984    58559     -425     
============================================
  Files          4741     4742       +1     
  Lines        280607   280629      +22     
  Branches      40894    40897       +3     
============================================
- Hits         199295   197871    -1424     
- Misses        65099    66290    +1191     
- Partials      16213    16468     +255

Impacted Files	Coverage Δ
...a/org/opensearch/test/OpenSearchIntegTestCase.java	`54.51% <0.00%> (-2.11%)`	⬇️
...pensearch/common/settings/FeatureFlagSettings.java	`50.00% <50.00%> (ø)`
.../java/org/opensearch/common/util/FeatureFlags.java	`80.00% <88.88%> (+30.00%)`	⬆️
server/src/main/java/org/opensearch/Version.java	`83.40% <100.00%> (+0.07%)`	⬆️
...org/opensearch/common/settings/SettingsModule.java	`84.45% <100.00%> (+0.32%)`	⬆️
server/src/main/java/org/opensearch/node/Node.java	`83.33% <100.00%> (+0.02%)`	⬆️
...n/indices/forcemerge/ForceMergeRequestBuilder.java	`0.00% <0.00%> (-75.00%)`	⬇️
.../indices/forcemerge/TransportForceMergeAction.java	`25.00% <0.00%> (-75.00%)`	⬇️
...port/ResponseHandlerFailureTransportException.java	`0.00% <0.00%> (-60.00%)`	⬇️
...a/org/opensearch/client/cluster/SniffModeInfo.java	`0.00% <0.00%> (-56.25%)`	⬇️
... and 486 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-01-08T11:43:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9016/
CommitID: 774b678
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T11:50:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9017/
CommitID: d4e80b8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T12:01:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9018/
CommitID: fbf43db
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T12:25:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9019/
CommitID: cd4bab2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T12:56:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9021/
CommitID: 7ddf790
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T13:19:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9022/
CommitID: b3ee17d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-08T14:15:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion

URL: https://build.ci.opensearch.org/job/gradle-check/9023/
CommitID: b5084ee
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

dreamer-89 · 2023-01-09T20:18:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕

TEST FAILURES:
      2 org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion
URL: https://build.ci.opensearch.org/job/gradle-check/9023/

CommitID: b5084ee
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Another segment replication flaky test. There are multiple test failures which is worrysome. @tan31989 Let's wait on this PR until we identify cause of this flakyness. We are priortizing fixing these tests as part of #5669. Please feel free to pick some if you are interested.

ntantri · 2023-01-10T03:29:19Z

Another segment replication flaky test. There are multiple test failures which is worrysome. @tan31989 Let's wait on this PR until we identify cause of this flakyness. We are priortizing fixing these tests as part of #5669. Please feel free to pick some if you are interested.

Yes, that makes sense. I believe if there are any fixes to follow up on, I will keep an eye open for merge conflicts if any.

dreamer-89 · 2023-01-19T05:48:14Z

Another segment replication flaky test. There are multiple test failures which is worrysome. @tan31989 Let's wait on this PR until we identify cause of this flakyness. We are priortizing fixing these tests as part of #5669. Please feel free to pick some if you are interested.

Yes, that makes sense. I believe if there are any fixes to follow up on, I will keep an eye open for merge conflicts if any.

Hi @tan31989, thank you for helping on this issue. We analyzed the flaky tests, most of them are failing due to doc count mismatch. There is a PR on the way to fix this. I think we can go ahead and merge this PR. Can you please resolve the merge conflicts and rebase your changes.

…#4959) This change introduces a static store "settings" in FeatureFlags.java file to enable isEnabled method to fetch flag settings defined in opensearch.yml. Signed-off-by: Nagaraj Tantri <[email protected]> (cherry picked from commit 241bd42)

github-actions · 2023-01-19T11:55:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9589/
CommitID: 272edee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-19T19:01:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/9589/

CommitID: 272edee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Segrep test failure testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion. This is already tracked in #5669. Refiring the gradle check!

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion" -Dtests.seed=692E412BA80E078D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-NI -Dtests.timezone=America/Kentucky/Monticello -Druntime.java=17

org.opensearch.snapshots.SegmentReplicationSnapshotIT > testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([692E412BA80E078D:46A1E74C1F5EF990]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1007)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:938)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:927)
        at org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion(SegmentReplicationSnapshotIT.java:180)

kotwanikunal · 2023-01-19T19:59:55Z

CHANGELOG.md

+- Support for labels on version bump PRs, skip label support for changelog verifier ([#4391](https://github.com/opensearch-project/OpenSearch/pull/4391))
+- Add a new node role 'search' which is dedicated to provide search capability ([#4689](https://github.com/opensearch-project/OpenSearch/pull/4689))
+- Introduce experimental searchable snapshot API ([#4680](https://github.com/opensearch-project/OpenSearch/pull/4680))
+- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))


Please add only your change within the changelog.

Suggested change

- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))

- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))

FYI @mch2 and @dreamer-89, the reason cherry-pick got so confused by this changelog entry is that it was added to the wrong section on the main branch. Since the changed was destined for a 2.x release, the changelog entry should have gone in the [Unreleased 2.x] section.

andrross · 2023-01-19T21:14:49Z

Thanks @tan31989! This backport is pretty messy for a couple different reasons, so I went ahead and created another attempt here: #5941

kotwanikunal · 2023-01-19T22:09:46Z

#5941 has been merged in to 2.x. Thanks for your contribution @tan31989!
I am closing this PR. Please re-open if there is any change pending.

ntantri requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and xuezhou25 as code owners January 5, 2023 07:51

ntantri force-pushed the backport/backport-4959-to-2.x branch from f1aee56 to a6d94fe Compare January 5, 2023 07:55

ntantri mentioned this pull request Jan 5, 2023

Added support for feature flags in opensearch.yml #4959

Merged

6 tasks

ntantri force-pushed the backport/backport-4959-to-2.x branch from a6d94fe to b296a89 Compare January 5, 2023 08:45

dreamer-89 requested changes Jan 5, 2023

View reviewed changes

ntantri force-pushed the backport/backport-4959-to-2.x branch from b296a89 to 51e344e Compare January 7, 2023 13:37

ntantri requested a review from VachaShah as a code owner January 7, 2023 13:37

ntantri force-pushed the backport/backport-4959-to-2.x branch from 51e344e to 523cd72 Compare January 7, 2023 13:44

ntantri force-pushed the backport/backport-4959-to-2.x branch from 523cd72 to 2e93b82 Compare January 8, 2023 04:28

ntantri force-pushed the backport/backport-4959-to-2.x branch 3 times, most recently from d4e80b8 to fbf43db Compare January 8, 2023 11:40

ntantri force-pushed the backport/backport-4959-to-2.x branch from fbf43db to cd4bab2 Compare January 8, 2023 12:08

ntantri force-pushed the backport/backport-4959-to-2.x branch from cd4bab2 to 7ddf790 Compare January 8, 2023 12:31

ntantri force-pushed the backport/backport-4959-to-2.x branch from 7ddf790 to b3ee17d Compare January 8, 2023 13:02

ntantri force-pushed the backport/backport-4959-to-2.x branch from b3ee17d to b5084ee Compare January 8, 2023 13:30

ntantri force-pushed the backport/backport-4959-to-2.x branch from b5084ee to 272edee Compare January 19, 2023 11:27

ntantri requested a review from gbbafna as a code owner January 19, 2023 11:27

kotwanikunal requested changes Jan 19, 2023

View reviewed changes

kotwanikunal mentioned this pull request Jan 19, 2023

[Backport 2.x] Enable SearchableSnapshotIT to run during build #5939

Merged

kotwanikunal closed this Jan 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for feature flags in opensearch.yml (#4959) #5707

Added support for feature flags in opensearch.yml (#4959) #5707

ntantri commented Jan 5, 2023

github-actions bot commented Jan 5, 2023

github-actions bot commented Jan 5, 2023

github-actions bot commented Jan 5, 2023

dreamer-89 Jan 5, 2023

ntantri Jan 7, 2023 •

edited

Loading

ntantri Jan 8, 2023

dreamer-89 Jan 8, 2023

github-actions bot commented Jan 8, 2023

codecov-commenter commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

github-actions bot commented Jan 8, 2023

dreamer-89 commented Jan 9, 2023 •

edited

Loading

Gradle Check (Jenkins) Run Completed with:

ntantri commented Jan 10, 2023

dreamer-89 commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

dreamer-89 commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal Jan 19, 2023

andrross Jan 19, 2023

andrross commented Jan 19, 2023

kotwanikunal commented Jan 19, 2023

	- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))
	- Added support for feature flags in opensearch.yml ([#4959](https://github.com/opensearch-project/OpenSearch/pull/4959))

Added support for feature flags in opensearch.yml (#4959) #5707

Added support for feature flags in opensearch.yml (#4959) #5707

Conversation

ntantri commented Jan 5, 2023

Description

Issues Resolved

Check List

github-actions bot commented Jan 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 5, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 Jan 5, 2023

Choose a reason for hiding this comment

ntantri Jan 7, 2023 • edited Loading

Choose a reason for hiding this comment

ntantri Jan 8, 2023

Choose a reason for hiding this comment

dreamer-89 Jan 8, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Jan 8, 2023

Codecov Report

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 8, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 9, 2023 • edited Loading

Gradle Check (Jenkins) Run Completed with:

ntantri commented Jan 10, 2023

dreamer-89 commented Jan 19, 2023

github-actions bot commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 19, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal Jan 19, 2023

Choose a reason for hiding this comment

andrross Jan 19, 2023

Choose a reason for hiding this comment

andrross commented Jan 19, 2023

kotwanikunal commented Jan 19, 2023

ntantri Jan 7, 2023 •

edited

Loading

dreamer-89 commented Jan 9, 2023 •

edited

Loading