Skip to content

Commit

Permalink
merge main
Browse files Browse the repository at this point in the history
Signed-off-by: Gao Binlong <[email protected]>
  • Loading branch information
gaobinlong committed Mar 26, 2024
2 parents 3a18815 + 3907ec9 commit 60ef6a1
Show file tree
Hide file tree
Showing 56 changed files with 952 additions and 161 deletions.
7 changes: 6 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Remote reindex: Add support for configurable retry mechanism ([#12561](https://github.com/opensearch-project/OpenSearch/pull/12561))
- [Admission Control] Integrate IO Usage Tracker to the Resource Usage Collector Service and Emit IO Usage Stats ([#11880](https://github.com/opensearch-project/OpenSearch/pull/11880))
- Tracing for deep search path ([#12103](https://github.com/opensearch-project/OpenSearch/pull/12103))
- Add explicit dependency to validatePom and generatePom tasks ([#12103](https://github.com/opensearch-project/OpenSearch/pull/12807))
- Add explicit dependency to validatePom and generatePom tasks ([#12807](https://github.com/opensearch-project/OpenSearch/pull/12807))
- Replace configureEach with all for publication iteration ([#12876](https://github.com/opensearch-project/OpenSearch/pull/12876))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand Down Expand Up @@ -104,6 +105,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Added

### Dependencies
- Bump `org.apache.commons:commons-configuration2` from 2.10.0 to 2.10.1 ([#12896](https://github.com/opensearch-project/OpenSearch/pull/12896))
- Bump `asm` from 9.6 to 9.7 ([#12908](https://github.com/opensearch-project/OpenSearch/pull/12908))
- Bump `net.minidev:json-smart` from 2.5.0 to 2.5.1 ([#12893](https://github.com/opensearch-project/OpenSearch/pull/12893))

### Changed
- [BWC and API enforcement] Enforcing the presence of API annotations at build time ([#12872](https://github.com/opensearch-project/OpenSearch/pull/12872))
Expand All @@ -114,6 +118,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

### Fixed
- Fix bulk API ignores ingest pipeline for upsert ([#12883](https://github.com/opensearch-project/OpenSearch/pull/12883))
- Fix issue with feature flags where default value may not be honored ([#12849](https://github.com/opensearch-project/OpenSearch/pull/12849))

### Security

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
[![Security Vulnerabilities](https://img.shields.io/github/issues/opensearch-project/OpenSearch/security%20vulnerability?labelColor=red)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"security%20vulnerability")
[![Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch)](https://github.com/opensearch-project/OpenSearch/issues)
[![Open Pull Requests](https://img.shields.io/github/issues-pr/opensearch-project/OpenSearch)](https://github.com/opensearch-project/OpenSearch/pulls)
[![2.10 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v2.10.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v2.10.0")
[![3.0 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v3.0.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v3.0.0")
[![2.14.0 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v2.14.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v2.14.0")
[![3.0.0 Open Issues](https://img.shields.io/github/issues/opensearch-project/OpenSearch/v3.0.0)](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+label%3A"v3.0.0")
[![GHA gradle check](https://github.com/opensearch-project/OpenSearch/actions/workflows/gradle-check.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/gradle-check.yml)
[![GHA validate pull request](https://github.com/opensearch-project/OpenSearch/actions/workflows/wrapper.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/wrapper.yml)
[![GHA precommit](https://github.com/opensearch-project/OpenSearch/actions/workflows/precommit.yml/badge.svg)](https://github.com/opensearch-project/OpenSearch/actions/workflows/precommit.yml)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ public class PomValidationPrecommitPlugin extends PrecommitPlugin {
public TaskProvider<? extends Task> createTask(Project project) {
TaskProvider<Task> validatePom = project.getTasks().register("validatePom");
PublishingExtension publishing = project.getExtensions().getByType(PublishingExtension.class);
publishing.getPublications().configureEach(publication -> {
publishing.getPublications().all(publication -> {
String publicationName = Util.capitalize(publication.getName());
TaskProvider<PomValidationTask> validateTask = project.getTasks()
.register("validate" + publicationName + "Pom", PomValidationTask.class);
Expand All @@ -59,7 +59,7 @@ public TaskProvider<? extends Task> createTask(Project project) {
validateTask.configure(task -> {
task.dependsOn(generateMavenPom);
task.getPomFile().fileProvider(generateMavenPom.map(GenerateMavenPom::getDestination));
publishing.getPublications().configureEach(publicationForPomGen -> {
publishing.getPublications().all(publicationForPomGen -> {
task.mustRunAfter(
project.getTasks()
.withType(GenerateMavenPom.class)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ public class GradleThreadsFilter implements ThreadFilter {
public boolean reject(Thread t) {
return t.getName().startsWith("Exec process")
|| t.getName().startsWith("Memory manager")
|| t.getName().startsWith("File watcher consumer");
|| t.getName().startsWith("File watcher consumer")
|| t.getName().startsWith("sshd-SshClient") /* Started by SshClient (sshd-core), part of SftpFileSystemProvider */
|| t.getName().startsWith("Thread-"); /* Started by AbstractFactoryManager (sshd-core), part of SftpFileSystemProvider */
}
}
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ icu4j = 70.1
supercsv = 2.4.0
log4j = 2.21.0
slf4j = 1.7.36
asm = 9.6
asm = 9.7
jettison = 1.5.4
woodstox = 6.4.0
kotlin = 1.7.10
Expand Down
4 changes: 2 additions & 2 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.6-all.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.7-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionSha256Sum=85719317abd2112f021d4f41f09ec370534ba288432065f4b477b6a3b652910d
distributionSha256Sum=194717442575a6f96e1c1befa2c30e9a4fc90f701d7aee33eb879b79e7ff05c0
1 change: 0 additions & 1 deletion modules/lang-expression/licenses/asm-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-expression/licenses/asm-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
073d7b3086e14beb604ced229c302feff6449723
1 change: 0 additions & 1 deletion modules/lang-expression/licenses/asm-commons-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-expression/licenses/asm-commons-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e86dda4696d3c185fcc95d8d311904e7ce38a53f
1 change: 0 additions & 1 deletion modules/lang-expression/licenses/asm-tree-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-expression/licenses/asm-tree-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e446a17b175bfb733b87c5c2560ccb4e57d69f1a
1 change: 0 additions & 1 deletion modules/lang-painless/licenses/asm-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-painless/licenses/asm-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
073d7b3086e14beb604ced229c302feff6449723
1 change: 0 additions & 1 deletion modules/lang-painless/licenses/asm-analysis-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-painless/licenses/asm-analysis-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e4a258b7eb96107106c0599f0061cfc1832fe07a
1 change: 0 additions & 1 deletion modules/lang-painless/licenses/asm-commons-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-painless/licenses/asm-commons-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e86dda4696d3c185fcc95d8d311904e7ce38a53f
1 change: 0 additions & 1 deletion modules/lang-painless/licenses/asm-tree-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-painless/licenses/asm-tree-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e446a17b175bfb733b87c5c2560ccb4e57d69f1a
1 change: 0 additions & 1 deletion modules/lang-painless/licenses/asm-util-9.6.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions modules/lang-painless/licenses/asm-util-9.7.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
c0655519f24d92af2202cb681cd7c1569df6ead6
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
- do:
scripts_painless_context: {}
- match: { contexts.0: aggregation_selector}
- match: { contexts.23: update}
- match: { contexts.24: update}
---

"Action to get all API values for score context":
Expand Down
2 changes: 1 addition & 1 deletion plugins/repository-hdfs/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ dependencies {
api "commons-codec:commons-codec:${versions.commonscodec}"
api 'commons-collections:commons-collections:3.2.2'
api "org.apache.commons:commons-compress:${versions.commonscompress}"
api 'org.apache.commons:commons-configuration2:2.10.0'
api 'org.apache.commons:commons-configuration2:2.10.1'
api 'commons-io:commons-io:2.15.1'
api 'org.apache.commons:commons-lang3:3.14.0'
implementation 'com.google.re2j:re2j:1.7'
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2b681b3bcddeaa5bf5c2a2939cd77e2f9ad6efda
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,8 @@
import org.opensearch.index.mapper.MapperService;
import org.opensearch.index.mapper.MapperService.MergeReason;
import org.opensearch.index.query.QueryShardContext;
import org.opensearch.index.remote.RemoteStorePathResolver;
import org.opensearch.index.remote.RemoteStorePathType;
import org.opensearch.index.remote.RemoteStorePathTypeResolver;
import org.opensearch.index.shard.IndexSettingProvider;
import org.opensearch.index.translog.Translog;
import org.opensearch.indices.IndexCreationException;
Expand All @@ -113,6 +113,7 @@
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
Expand Down Expand Up @@ -170,7 +171,7 @@ public class MetadataCreateIndexService {
private AwarenessReplicaBalance awarenessReplicaBalance;

@Nullable
private final RemoteStorePathResolver remoteStorePathResolver;
private final RemoteStorePathTypeResolver remoteStorePathTypeResolver;

public MetadataCreateIndexService(
final Settings settings,
Expand Down Expand Up @@ -203,8 +204,8 @@ public MetadataCreateIndexService(

// Task is onboarded for throttling, it will get retried from associated TransportClusterManagerNodeAction.
createIndexTaskKey = clusterService.registerClusterManagerTask(ClusterManagerTaskKeys.CREATE_INDEX_KEY, true);
remoteStorePathResolver = isRemoteDataAttributePresent(settings)
? new RemoteStorePathResolver(clusterService.getClusterSettings())
remoteStorePathTypeResolver = isRemoteDataAttributePresent(settings)
? new RemoteStorePathTypeResolver(clusterService.getClusterSettings())
: null;
}

Expand Down Expand Up @@ -553,7 +554,7 @@ IndexMetadata buildAndValidateTemporaryIndexMetadata(
tmpImdBuilder.setRoutingNumShards(routingNumShards);
tmpImdBuilder.settings(indexSettings);
tmpImdBuilder.system(isSystem);
addRemoteCustomData(tmpImdBuilder);
addRemoteStorePathTypeInCustomData(tmpImdBuilder, true);

// Set up everything, now locally create the index to see that things are ok, and apply
IndexMetadata tempMetadata = tmpImdBuilder.build();
Expand All @@ -562,17 +563,24 @@ IndexMetadata buildAndValidateTemporaryIndexMetadata(
return tempMetadata;
}

public void addRemoteCustomData(IndexMetadata.Builder tmpImdBuilder) {
if (remoteStorePathResolver != null) {
/**
* Adds the remote store path type information in custom data of index metadata.
*
* @param tmpImdBuilder index metadata builder.
* @param assertNullOldType flag to verify that the old remote store path type is null
*/
public void addRemoteStorePathTypeInCustomData(IndexMetadata.Builder tmpImdBuilder, boolean assertNullOldType) {
if (remoteStorePathTypeResolver != null) {
// It is possible that remote custom data exists already. In such cases, we need to only update the path type
// in the remote store custom data map.
Map<String, String> existingRemoteCustomData = tmpImdBuilder.removeCustom(IndexMetadata.REMOTE_STORE_CUSTOM_KEY);
Map<String, String> remoteCustomData = existingRemoteCustomData == null
? new HashMap<>()
: new HashMap<>(existingRemoteCustomData);
// Determine the path type for use using the remoteStorePathResolver.
String newPathType = remoteStorePathResolver.resolveType().toString();
String newPathType = remoteStorePathTypeResolver.getType().toString();
String oldPathType = remoteCustomData.put(RemoteStorePathType.NAME, newPathType);
assert !assertNullOldType || Objects.isNull(oldPathType);
logger.trace(() -> new ParameterizedMessage("Added new path type {}, replaced old path type {}", newPathType, oldPathType));
tmpImdBuilder.putCustom(IndexMetadata.REMOTE_STORE_CUSTOM_KEY, remoteCustomData);
}
Expand Down
82 changes: 54 additions & 28 deletions server/src/main/java/org/opensearch/common/util/FeatureFlags.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@
import org.opensearch.common.settings.Setting.Property;
import org.opensearch.common.settings.Settings;

import java.util.List;

/**
* Utility class to manage feature flags. Feature flags are system properties that must be set on the JVM.
* These are used to gate the visibility/availability of incomplete features. Fore more information, see
* These are used to gate the visibility/availability of incomplete features. For more information, see
* https://featureflags.io/feature-flag-introduction/
*
* @opensearch.internal
Expand Down Expand Up @@ -65,19 +67,69 @@ public class FeatureFlags {
*/
public static final String PLUGGABLE_CACHE = "opensearch.experimental.feature.pluggable.caching.enabled";

public static final Setting<Boolean> REMOTE_STORE_MIGRATION_EXPERIMENTAL_SETTING = Setting.boolSetting(
REMOTE_STORE_MIGRATION_EXPERIMENTAL,
false,
Property.NodeScope
);

public static final Setting<Boolean> EXTENSIONS_SETTING = Setting.boolSetting(EXTENSIONS, false, Property.NodeScope);

public static final Setting<Boolean> IDENTITY_SETTING = Setting.boolSetting(IDENTITY, false, Property.NodeScope);

public static final Setting<Boolean> TELEMETRY_SETTING = Setting.boolSetting(TELEMETRY, false, Property.NodeScope);

public static final Setting<Boolean> DATETIME_FORMATTER_CACHING_SETTING = Setting.boolSetting(
DATETIME_FORMATTER_CACHING,
true,
Property.NodeScope
);

public static final Setting<Boolean> WRITEABLE_REMOTE_INDEX_SETTING = Setting.boolSetting(
WRITEABLE_REMOTE_INDEX,
false,
Property.NodeScope
);

public static final Setting<Boolean> PLUGGABLE_CACHE_SETTING = Setting.boolSetting(PLUGGABLE_CACHE, false, Property.NodeScope);

private static final List<Setting<Boolean>> ALL_FEATURE_FLAG_SETTINGS = List.of(
REMOTE_STORE_MIGRATION_EXPERIMENTAL_SETTING,
EXTENSIONS_SETTING,
IDENTITY_SETTING,
TELEMETRY_SETTING,
DATETIME_FORMATTER_CACHING_SETTING,
WRITEABLE_REMOTE_INDEX_SETTING,
PLUGGABLE_CACHE_SETTING
);
/**
* Should store the settings from opensearch.yml.
*/
private static Settings settings;

static {
Settings.Builder settingsBuilder = Settings.builder();
for (Setting<Boolean> ffSetting : ALL_FEATURE_FLAG_SETTINGS) {
settingsBuilder = settingsBuilder.put(ffSetting.getKey(), ffSetting.getDefault(Settings.EMPTY));
}
settings = settingsBuilder.build();
}

/**
* This method is responsible to map settings from opensearch.yml to local stored
* settings value. That is used for the existing isEnabled method.
*
* @param openSearchSettings The settings stored in opensearch.yml.
*/
public static void initializeFeatureFlags(Settings openSearchSettings) {
settings = openSearchSettings;
Settings.Builder settingsBuilder = Settings.builder();
for (Setting<Boolean> ffSetting : ALL_FEATURE_FLAG_SETTINGS) {
settingsBuilder = settingsBuilder.put(
ffSetting.getKey(),
openSearchSettings.getAsBoolean(ffSetting.getKey(), ffSetting.getDefault(openSearchSettings))
);
}
settings = settingsBuilder.build();
}

/**
Expand All @@ -103,30 +155,4 @@ public static boolean isEnabled(Setting<Boolean> featureFlag) {
return featureFlag.getDefault(Settings.EMPTY);
}
}

public static final Setting<Boolean> REMOTE_STORE_MIGRATION_EXPERIMENTAL_SETTING = Setting.boolSetting(
REMOTE_STORE_MIGRATION_EXPERIMENTAL,
false,
Property.NodeScope
);

public static final Setting<Boolean> EXTENSIONS_SETTING = Setting.boolSetting(EXTENSIONS, false, Property.NodeScope);

public static final Setting<Boolean> IDENTITY_SETTING = Setting.boolSetting(IDENTITY, false, Property.NodeScope);

public static final Setting<Boolean> TELEMETRY_SETTING = Setting.boolSetting(TELEMETRY, false, Property.NodeScope);

public static final Setting<Boolean> DATETIME_FORMATTER_CACHING_SETTING = Setting.boolSetting(
DATETIME_FORMATTER_CACHING,
true,
Property.NodeScope
);

public static final Setting<Boolean> WRITEABLE_REMOTE_INDEX_SETTING = Setting.boolSetting(
WRITEABLE_REMOTE_INDEX,
false,
Property.NodeScope
);

public static final Setting<Boolean> PLUGGABLE_CACHE_SETTING = Setting.boolSetting(PLUGGABLE_CACHE, false, Property.NodeScope);
}
3 changes: 2 additions & 1 deletion server/src/main/java/org/opensearch/index/IndexService.java
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,8 @@ public synchronized IndexShard createShard(
remoteDirectory = ((RemoteSegmentStoreDirectoryFactory) remoteDirectoryFactory).newDirectory(
RemoteStoreNodeAttribute.getRemoteStoreSegmentRepo(this.indexSettings.getNodeSettings()),
this.indexSettings.getUUID(),
shardId
shardId,
this.indexSettings.getRemoteStorePathType()
);
}
remoteStore = new Store(shardId, this.indexSettings, remoteDirectory, lock, Store.OnClose.EMPTY, path);
Expand Down
9 changes: 9 additions & 0 deletions server/src/main/java/org/opensearch/index/IndexSettings.java
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
import org.opensearch.core.common.unit.ByteSizeUnit;
import org.opensearch.core.common.unit.ByteSizeValue;
import org.opensearch.core.index.Index;
import org.opensearch.index.remote.RemoteStorePathType;
import org.opensearch.index.translog.Translog;
import org.opensearch.indices.replication.common.ReplicationType;
import org.opensearch.ingest.IngestService;
Expand All @@ -59,6 +60,7 @@
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import java.util.function.Consumer;
Expand Down Expand Up @@ -1905,4 +1907,11 @@ public double getDocIdFuzzySetFalsePositiveProbability() {
public void setDocIdFuzzySetFalsePositiveProbability(double docIdFuzzySetFalsePositiveProbability) {
this.docIdFuzzySetFalsePositiveProbability = docIdFuzzySetFalsePositiveProbability;
}

public RemoteStorePathType getRemoteStorePathType() {
Map<String, String> remoteCustomData = indexMetadata.getCustomData(IndexMetadata.REMOTE_STORE_CUSTOM_KEY);
return remoteCustomData != null && remoteCustomData.containsKey(RemoteStorePathType.NAME)
? RemoteStorePathType.parseString(remoteCustomData.get(RemoteStorePathType.NAME))
: RemoteStorePathType.FIXED;
}
}
Loading

0 comments on commit 60ef6a1

Please sign in to comment.