Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Treat . as a nested field in field_map of text embedding processor #488

Closed
wants to merge 5 commits into from
Closed

Treat . as a nested field in field_map of text embedding processor #488

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7006c97
Treat dot as nested field in field_map for inference processor
Sanjana679 Nov 8, 2023
2c7f491
Treat dot in field name as a nested field in field_map of text embedd…
Sanjana679 Nov 9, 2023
a5b1c4a
More meaningful name for new targetKey value
Sanjana679 Nov 12, 2023
0e38fea
Added validation for nested fields in field_map
Sanjana679 Nov 13, 2023
2177ba2
Merge branch 'opensearch-project:main' into feature/text-embedding-pr…
Sanjana679 Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,16 @@
for (Map.Entry<String, Object> fieldMapEntry : fieldMap.entrySet()) {
String originalKey = fieldMapEntry.getKey();
Object targetKey = fieldMapEntry.getValue();

int nestedDotIndex = originalKey.indexOf('.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a user-provided info, can we add basic validation if it's not already done as part of the processor/pipeline definition.

if multiple levels of nested fields are needed this code may need a rework

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some basic validation done in validateEmbeddingConfiguration which is run on fieldMap in the constructor in this file. However, I will add some extra validation for nested fields.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use a static constant and avoid magic characters in code.
e.g.

private static final char FIELD_SEPARATOR = '.';

if (nestedDotIndex != -1) {
Map<String, Object> temp = new LinkedHashMap<>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this map object? Can you please use more meaningful name for map variable

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a more meaningful name for the map variable.

temp.put(originalKey.substring(nestedDotIndex + 1), targetKey);
targetKey = temp;

Check warning on line 162 in src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java#L160-L162 

Added lines #L160 - L162 were not covered by tests

originalKey = originalKey.substring(0, nestedDotIndex);

Check warning on line 164 in src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java#L164 

Added line #L164 was not covered by tests
}

if (targetKey instanceof Map) {
Map<String, Object> treeRes = new LinkedHashMap<>();
buildMapWithProcessorKeyAndOriginalValueForMapType(originalKey, targetKey, sourceAndMetadataMap, treeRes);
Expand Down
Loading