Skip to content

Commit

Permalink
Upgrade BWC version and fix neural search build (opensearch-project#868)
Browse files Browse the repository at this point in the history
Signed-off-by: Varun Jain <[email protected]>
  • Loading branch information
vibrantvarun authored Aug 20, 2024
1 parent 30a99fb commit 03d528e
Show file tree
Hide file tree
Showing 7 changed files with 149 additions and 26 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/backwards_compatibility_tests_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
matrix:
java: [ 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version : ["2.9.0","2.10.0","2.11.0","2.12.0","2.13.0","2.14.0","2.15.0","2.16.0-SNAPSHOT"]
bwc_version : ["2.9.0","2.10.0","2.11.0","2.12.0","2.13.0","2.14.0","2.15.0","2.16.0","2.17.0-SNAPSHOT"]
opensearch_version : [ "3.0.0-SNAPSHOT" ]

name: NeuralSearch Restart-Upgrade BWC Tests
Expand All @@ -42,7 +42,7 @@ jobs:
matrix:
java: [ 21 ]
os: [ubuntu-latest,windows-latest]
bwc_version: [ "2.16.0-SNAPSHOT" ]
bwc_version: [ "2.17.0-SNAPSHOT" ]
opensearch_version: [ "3.0.0-SNAPSHOT" ]

name: NeuralSearch Rolling-Upgrade BWC Tests
Expand Down
22 changes: 11 additions & 11 deletions TRIAGING.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">

The maintainers of the k-NN/neural-search Repo's seek to promote an inclusive and engaged community of contributors. In
order to facilitate this, bi-weekly triage meetings are open-to-all and attendance is encouraged for anyone who hopes to
contribute, discuss an issue, or learn more about the project. To learn more about contributing to the
The maintainers of the k-NN/neural-search Repo's seek to promote an inclusive and engaged community of contributors. In
order to facilitate this, bi-weekly triage meetings are open-to-all and attendance is encouraged for anyone who hopes to
contribute, discuss an issue, or learn more about the project. To learn more about contributing to the
k-NN/neural-search Repo visit the [Contributing](./CONTRIBUTING.md) documentation.

### Do I need to attend for my issue to be addressed/triaged?
Expand All @@ -11,19 +11,19 @@ Attendance is not required for your issue to be triaged or addressed. All new is

### What happens if my issue does not get covered this time?

Each meeting we seek to address all new issues. However, should we run out of time before your issue is discussed, you
Each meeting we seek to address all new issues. However, should we run out of time before your issue is discussed, you
are always welcome to attend the next meeting or to follow up on the issue post itself.

### How do I join the Backlog & Triage meeting?

Meetings are hosted regularly at 5 PM Pacific Time on Tuesdays bi-weekly and can be joined via the links posted on the
[OpenSearch Meetup Group](https://www.meetup.com/opensearch/events/) list of events. The event will be titled
Meetings are hosted regularly at 5 PM Pacific Time on Tuesdays bi-weekly and can be joined via the links posted on the
[OpenSearch Meetup Group](https://www.meetup.com/opensearch/events/) list of events. The event will be titled
`Development Backlog & Triage Meeting - k-NN/neural-search`.

After joining the Chime meeting, you can enable your video / voice to join the discussion. If you do not have a webcam
After joining the Chime meeting, you can enable your video / voice to join the discussion. If you do not have a webcam
or microphone available, you can still join in via the text chat.

If you have an issue you'd like to bring forth please consider getting a link to the issue so it can be presented to
If you have an issue you'd like to bring forth please consider getting a link to the issue so it can be presented to
everyone in the meeting.

### Is there an agenda for each week?
Expand All @@ -48,13 +48,13 @@ No, all are welcome and encouraged to attend. Attending the Backlog & Triage mee

### What if I have an issue that is almost a duplicate, should I open a new one to be triaged?

You can always open an issue including one that you think may be a duplicate. However, in cases where you believe there
is an important distinction to be made between an existing issue and your newly created one, you are encouraged to
You can always open an issue including one that you think may be a duplicate. However, in cases where you believe there
is an important distinction to be made between an existing issue and your newly created one, you are encouraged to
attend the triaging meeting to explain.

### What if I have follow-up questions on an issue?

If you have an existing issue you would like to discuss, you can always comment on the issue itself. Alternatively, you
If you have an existing issue you would like to discuss, you can always comment on the issue itself. Alternatively, you
are welcome to come to the triage meeting to discuss.

### Is this meeting a good place to get help setting up k-NN/neural-search features on my OpenSearch instance?
Expand Down
4 changes: 2 additions & 2 deletions gradle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
# https://github.com/opensearch-project/OpenSearch/blob/main/libs/core/src/main/java/org/opensearch/Version.java .
# Wired compatibility of OpenSearch works like 3.x version is compatible with 2.(latest-major) version.
# Therefore, to run rolling-upgrade BWC Test on local machine the BWC version here should be set 2.(latest-major).
systemProp.bwc.version=2.16.0-SNAPSHOT
systemProp.bwc.bundle.version=2.16.0
systemProp.bwc.version=2.17.0-SNAPSHOT
systemProp.bwc.bundle.version=2.17.0

# For fixing Spotless check with Java 17
org.gradle.jvmargs=--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ private void createBasicKnnIndex() throws IOException {
.endObject()
.toString();
mapping = mapping.substring(1, mapping.length() - 1);
createIndex(KNN_INDEX_NAME, Settings.EMPTY, mapping);
Settings settings = Settings.builder().put("index.knn", true).build();
createIndex(KNN_INDEX_NAME, settings, mapping);
}

private Set<String> getAllInstalledPlugins() throws IOException {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,27 @@
import static org.opensearch.neuralsearch.query.NeuralQueryBuilder.MODEL_ID_FIELD;
import static org.opensearch.neuralsearch.query.NeuralQueryBuilder.QUERY_TEXT_FIELD;

import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;
import java.util.Optional;
import java.util.Set;
import java.util.function.Supplier;
import java.util.stream.Collectors;

import org.apache.lucene.search.MatchNoDocsQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;
import org.opensearch.Version;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.io.stream.BytesStreamOutput;
import org.opensearch.common.settings.ClusterSettings;
import org.opensearch.common.settings.Setting;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.xcontent.XContentFactory;
import org.opensearch.core.ParseField;
import org.opensearch.core.common.ParsingException;
Expand All @@ -47,8 +56,13 @@
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.index.query.QueryShardContext;
import org.opensearch.index.query.TermQueryBuilder;
import org.opensearch.knn.index.KNNSettings;
import org.opensearch.knn.index.SpaceType;
import org.opensearch.knn.index.VectorDataType;
import org.opensearch.knn.index.engine.KNNEngine;
import org.opensearch.knn.index.engine.KNNMethodContext;
import org.opensearch.knn.index.engine.MethodComponentContext;
import org.opensearch.knn.index.mapper.KNNMappingConfig;
import org.opensearch.knn.index.mapper.KNNVectorFieldType;
import org.opensearch.knn.index.query.KNNQuery;
import org.opensearch.knn.index.query.KNNQueryBuilder;
Expand All @@ -69,6 +83,26 @@ public class HybridQueryBuilderTests extends OpenSearchQueryTestCase {
static final float BOOST = 1.8f;
static final Supplier<float[]> TEST_VECTOR_SUPPLIER = () -> new float[4];
static final QueryBuilder TEST_FILTER = new MatchAllQueryBuilder();
@Mock
private ClusterService clusterService;
private AutoCloseable openMocks;

@Override
public void setUp() throws Exception {
super.setUp();
openMocks = MockitoAnnotations.openMocks(this);
// This is required to make sure that before every test we are initializing the KNNSettings. Not doing this
// leads to failures of unit tests cases when a unit test is run separately. Try running this test:
// ./gradlew ':test' --tests "org.opensearch.knn.training.TrainingJobTests.testRun_success" and see it fails
// but if run along with other tests this test passes.
initKNNSettings();
}

@Override
public void tearDown() throws Exception {
super.tearDown();
openMocks.close();
}

@SneakyThrows
public void testDoToQuery_whenNoSubqueries_thenBuildSuccessfully() {
Expand All @@ -86,11 +120,14 @@ public void testDoToQuery_whenOneSubquery_thenBuildSuccessfully() {
Index dummyIndex = new Index("dummy", "dummy");
QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
KNNMethodContext knnMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.of(knnMethodContext));
when(mockQueryShardContext.index()).thenReturn(dummyIndex);
when(mockKNNVectorField.getDimension()).thenReturn(4);
when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);

NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
.queryText(QUERY_TEXT)
Expand All @@ -116,10 +153,13 @@ public void testDoToQuery_whenMultipleSubqueries_thenBuildSuccessfully() {
Index dummyIndex = new Index("dummy", "dummy");
QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
KNNMethodContext knnMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.of(knnMethodContext));
when(mockQueryShardContext.index()).thenReturn(dummyIndex);
when(mockKNNVectorField.getDimension()).thenReturn(4);
when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);
when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);

NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
Expand Down Expand Up @@ -367,8 +407,10 @@ public void testToXContent_whenIncomingJsonIsCorrect_thenSuccessful() {
Index dummyIndex = new Index("dummy", "dummy");
QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
when(mockQueryShardContext.index()).thenReturn(dummyIndex);
when(mockKNNVectorField.getDimension()).thenReturn(4);
when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);

NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
Expand Down Expand Up @@ -584,9 +626,11 @@ public void testRewrite_whenMultipleSubQueries_thenReturnBuilderForEachSubQuery(

QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
Index dummyIndex = new Index("dummy", "dummy");
when(mockQueryShardContext.index()).thenReturn(dummyIndex);
when(mockKNNVectorField.getDimension()).thenReturn(4);
when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);

TextFieldMapper.TextFieldType fieldType = (TextFieldMapper.TextFieldType) createMapperService().fieldType(TEXT_FIELD_NAME);
Expand Down Expand Up @@ -737,4 +781,17 @@ private void setUpClusterService() {
ClusterService clusterService = NeuralSearchClusterTestUtils.mockClusterService(Version.CURRENT);
NeuralSearchClusterUtil.instance().initialize(clusterService);
}

private void initKNNSettings() {
Set<Setting<?>> defaultClusterSettings = new HashSet<>(ClusterSettings.BUILT_IN_CLUSTER_SETTINGS);
defaultClusterSettings.addAll(
KNNSettings.state()
.getSettings()
.stream()
.filter(s -> s.getProperties().contains(Setting.Property.NodeScope))
.collect(Collectors.toList())
);
when(clusterService.getClusterSettings()).thenReturn(new ClusterSettings(Settings.EMPTY, defaultClusterSettings));
KNNSettings.state().setClusterService(clusterService);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,11 @@

import java.io.IOException;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;

import org.apache.lucene.document.FieldType;
Expand All @@ -33,13 +36,24 @@
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.analysis.MockAnalyzer;
import org.apache.lucene.tests.search.QueryUtils;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;
import org.opensearch.cluster.service.ClusterService;
import org.opensearch.common.settings.ClusterSettings;
import org.opensearch.common.settings.Setting;
import org.opensearch.common.settings.Settings;
import org.opensearch.core.index.Index;
import org.opensearch.index.mapper.TextFieldMapper;
import org.opensearch.index.query.BoolQueryBuilder;
import org.opensearch.index.query.QueryBuilders;
import org.opensearch.index.query.QueryShardContext;
import org.opensearch.knn.index.KNNSettings;
import org.opensearch.knn.index.SpaceType;
import org.opensearch.knn.index.VectorDataType;
import org.opensearch.knn.index.engine.KNNEngine;
import org.opensearch.knn.index.engine.KNNMethodContext;
import org.opensearch.knn.index.engine.MethodComponentContext;
import org.opensearch.knn.index.mapper.KNNMappingConfig;
import org.opensearch.knn.index.mapper.KNNVectorFieldType;
import org.opensearch.knn.index.query.KNNQueryBuilder;

Expand All @@ -54,6 +68,26 @@ public class HybridQueryTests extends OpenSearchQueryTestCase {
static final String TERM_ANOTHER_QUERY_TEXT = "anotherkeyword";
static final float[] VECTOR_QUERY = new float[] { 1.0f, 2.0f, 2.1f, 0.6f };
static final int K = 2;
@Mock
protected ClusterService clusterService;
private AutoCloseable openMocks;

@Override
public void setUp() throws Exception {
super.setUp();
openMocks = MockitoAnnotations.openMocks(this);
// This is required to make sure that before every test we are initializing the KNNSettings. Not doing this
// leads to failures of unit tests cases when a unit test is run separately. Try running this test:
// ./gradlew ':test' --tests "org.opensearch.knn.training.TrainingJobTests.testRun_success" and see it fails
// but if run along with other tests this test passes.
initKNNSettings();
}

@Override
public void tearDown() throws Exception {
super.tearDown();
openMocks.close();
}

@SneakyThrows
public void testQueryBasics_whenMultipleDifferentQueries_thenSuccessful() {
Expand Down Expand Up @@ -116,10 +150,13 @@ public void testRewrite_whenRewriteQuery_thenSuccessful() {

Index dummyIndex = new Index("dummy", "dummy");
KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
KNNMethodContext mockKNNMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.ofNullable(mockKNNMethodContext));
when(mockQueryShardContext.index()).thenReturn(dummyIndex);
when(mockKNNVectorField.getDimension()).thenReturn(4);
when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);
when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
KNNQueryBuilder knnQueryBuilder = new KNNQueryBuilder(VECTOR_FIELD_NAME, VECTOR_QUERY, K);
Query knnQuery = knnQueryBuilder.toQuery(mockQueryShardContext);
Expand Down Expand Up @@ -316,4 +353,17 @@ public void testFilter_whenSubQueriesWithFilterPassed_thenSuccessful() {
}
assertEquals(2, countOfQueries);
}

private void initKNNSettings() {
Set<Setting<?>> defaultClusterSettings = new HashSet<>(ClusterSettings.BUILT_IN_CLUSTER_SETTINGS);
defaultClusterSettings.addAll(
KNNSettings.state()
.getSettings()
.stream()
.filter(s -> s.getProperties().contains(Setting.Property.NodeScope))
.collect(Collectors.toList())
);
when(clusterService.getClusterSettings()).thenReturn(new ClusterSettings(Settings.EMPTY, defaultClusterSettings));
KNNSettings.state().setClusterService(clusterService);
}
}
Loading

0 comments on commit 03d528e

Please sign in to comment.