[BUG] OpenSearch 2.1.0 SNAPSHOT recent changes fail security plugin integration tests #3554

cliu123 · 2022-06-09T19:16:01Z

Describe the bug
JSON is not parsed properly by Jackson.

To Reproduce

The failing GHA: https://github.com/opensearch-project/security/runs/6807080046?check_suite_focus=true
One of the failing tests: https://github.com/opensearch-project/security/blob/f431ec2201e1466b7c12528347a1f54cf64387c9/src/test/java/org/opensearch/security/dlic/rest/api/RolesMappingApiTest.java#L51
Running the test reproduces the issue.

Plugins
Security plugin

Additional context
Stacktrace:

[2022-06-08T23:32:20,280][ERROR][org.opensearch.security.dlic.rest.validation.RolesMappingValidator] BODY_NOT_PARSEABLE
2022-06-08T23:32:20.3699823Z     com.fasterxml.jackson.core.JsonParseException: Unexpected character ('"' (code 34)): was expecting comma to separate Object entries
2022-06-08T23:32:20.3700239Z      at [Source: (String)"{
2022-06-08T23:32:20.3700469Z       "users": [
2022-06-08T23:32:20.3700677Z         "sisko",
2022-06-08T23:32:20.3700876Z         "janeway",
2022-06-08T23:32:20.3701078Z         "kirk"
2022-06-08T23:32:20.3701253Z       ]
2022-06-08T23:32:20.3701455Z       "backend_roles": [
2022-06-08T23:32:20.3701672Z         "captains",
2022-06-08T23:32:20.3701873Z         "role2",
2022-06-08T23:32:20.3702072Z         "role3"
2022-06-08T23:32:20.3702263Z       ]
2022-06-08T23:32:20.3702549Z       "hosts": [
2022-06-08T23:32:20.3702736Z         "8.8.8.8",
2022-06-08T23:32:20.3702930Z         "8.8.4.4"
2022-06-08T23:32:20.3703121Z       ]
2022-06-08T23:32:20.3703298Z     }
2022-06-08T23:32:20.3703493Z     "; line: 7, column: 4]
2022-06-08T23:32:20.3704031Z     	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391) ~[jackson-core-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3705158Z     	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735) ~[jackson-core-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3705970Z     	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659) ~[jackson-core-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3706803Z     	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipComma(ReaderBasedJsonParser.java:2382) ~[jackson-core-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3707781Z     	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:947) ~[jackson-core-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3708774Z     	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:437) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3709749Z     	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:84) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3710712Z     	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:20) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3711746Z     	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3712684Z     	at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4716) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3713443Z     	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3076) ~[jackson-databind-2.13.2.jar:2.13.2]
2022-06-08T23:32:20.3714023Z     	at org.opensearch.security.DefaultObjectMapper$3.run(DefaultObjectMapper.java:154) ~[main/:?]
2022-06-08T23:32:20.3714552Z     	at org.opensearch.security.DefaultObjectMapper$3.run(DefaultObjectMapper.java:151) ~[main/:?]
2022-06-08T23:32:20.3715066Z     	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
2022-06-08T23:32:20.3715604Z     	at org.opensearch.security.DefaultObjectMapper.readTree(DefaultObjectMapper.java:151) ~[main/:?]
2022-06-08T23:32:20.3716339Z     	at org.opensearch.security.dlic.rest.validation.AbstractConfigurationValidator.validate(AbstractConfigurationValidator.java:128) ~[main/:?]
2022-06-08T23:32:20.3717135Z     	at org.opensearch.security.dlic.rest.api.AbstractApiAction.handleApiRequest(AbstractApiAction.java:111) ~[main/:?]
2022-06-08T23:32:20.3717923Z     	at org.opensearch.security.dlic.rest.api.PatchableResourceApiAction.handleApiRequest(PatchableResourceApiAction.java:268) ~[main/:?]
2022-06-08T23:32:20.3718687Z     	at org.opensearch.security.dlic.rest.api.AbstractApiAction.lambda$prepareRequest$2(AbstractApiAction.java:401) ~[main/:?]
2022-06-08T23:32:20.3719245Z     	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
2022-06-08T23:32:20.3719673Z     	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
2022-06-08T23:32:20.3720431Z     	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:739) [opensearch-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
2022-06-08T23:32:20.3721069Z     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
2022-06-08T23:32:20.3721571Z     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
2022-06-08T23:32:20.3721971Z     	at java.lang.Thread.run(Thread.java:833) [?:?]

The text was updated successfully, but these errors were encountered:

cliu123 · 2022-06-09T19:53:29Z

Security plugin has to downgrade to OpenSearch 2.0.0 because of this issue: opensearch-project/security#1882

reta · 2022-06-09T20:06:02Z

@tlfeng I think you know what it is :), could be related to ab478ba ?

tlfeng · 2022-06-09T20:29:00Z

could be related to ab478ba

@reta 😄 It's not likely related to the parsing change in Node Sniffer, because the change is limited to the Sniffer, and Sniffer is a separate library based on Rest Client, which is not used by security plugin. All json format data is parsed by fasterxml.jackson in the code.

cliu123 · 2022-06-09T20:55:53Z

@reta @tlfeng Thanks for checking the issue! Security plugin has downgraded to OpenSearch 2.1.0 where this issue doesn't exist. Please let us know when the root cause gets identified and the issue gets fixed. We're happy to verify the fix.

xuezhou25 · 2022-06-09T22:11:37Z

👀

xuezhou25 · 2022-06-09T23:25:41Z

In my point of view, the error JsonParseException is not the fail reason because there is the same error in 2.0.0.
The actual cause for the test failure is PR#3524, commit fc8803f.
In create document API:
In version 2.0.1 and above, "type" parameter is accepted, but will be ignored.
In version 2.0.0, "type" parmater is not accepted.
The failure occurs in this test case, and maybe it should be adjusted for OpenSearch 2.0.1 and 2.1.0 .
https://github.com/opensearch-project/security/blob/2.0.0.0/src/test/java/org/opensearch/security/dlic/rest/api/RolesMappingApiTest.java#L344-L345

The error message for the actual test failure is:

java.lang.AssertionError: expected:<400> but was:<200>
    at org.junit.Assert.fail(Assert.java:89)
    at org.junit.Assert.failNotEquals(Assert.java:835)
    at org.junit.Assert.assertEquals(Assert.java:647)
    at org.junit.Assert.assertEquals(Assert.java:633)
    at org.opensearch.security.dlic.rest.api.AbstractRestApiUnitTest.checkWriteAccess(AbstractRestApiUnitTest.java:226)
    at org.opensearch.security.dlic.rest.api.RolesMappingApiTest.checkAllSfAllowed(RolesMappingApiTest.java:345)
    at org.opensearch.security.dlic.rest.api.RolesMappingApiTest.testRolesMappingApi(RolesMappingApiTest.java:308)
...

cliu123 · 2022-06-10T22:06:08Z

Thanks @xuezhou25 @dreamer-89 @tlfeng for helping with this issue!

tlfeng · 2022-06-10T22:55:58Z

Draw a conclusion here, looks like security plugin have to do nothing with the test, since the behavior of "type" parameter in the REST API is changed back to the same as 2.0.0 in the commit 6462a54, "type" is not allowed as a parameter.

cliu123 · 2022-06-13T07:01:27Z

Draw a conclusion here, looks like security plugin have to do nothing with the test, since the behavior of "type" parameter in the REST API is changed back to the same as 2.0.0 in the commit 6462a54, "type" is not allowed as a parameter.

A quick clarification on the conclusion: The fix in security plugin tests is a part of the version bump PR.

cliu123 added bug Something isn't working untriaged labels Jun 9, 2022

zelinh mentioned this issue Jun 10, 2022

Add more components to 2.0.1 manifest and test manifests opensearch-project/opensearch-build#2178

Merged

cliu123 closed this as completed Jun 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] OpenSearch 2.1.0 SNAPSHOT recent changes fail security plugin integration tests #3554

[BUG] OpenSearch 2.1.0 SNAPSHOT recent changes fail security plugin integration tests #3554

cliu123 commented Jun 9, 2022

cliu123 commented Jun 9, 2022

reta commented Jun 9, 2022 •

edited

Loading

tlfeng commented Jun 9, 2022

cliu123 commented Jun 9, 2022

xuezhou25 commented Jun 9, 2022

xuezhou25 commented Jun 9, 2022 •

edited

Loading

cliu123 commented Jun 10, 2022 •

edited

Loading

tlfeng commented Jun 10, 2022

cliu123 commented Jun 13, 2022 •

edited

Loading

[BUG] OpenSearch 2.1.0 SNAPSHOT recent changes fail security plugin integration tests #3554

[BUG] OpenSearch 2.1.0 SNAPSHOT recent changes fail security plugin integration tests #3554

Comments

cliu123 commented Jun 9, 2022

cliu123 commented Jun 9, 2022

reta commented Jun 9, 2022 • edited Loading

tlfeng commented Jun 9, 2022

cliu123 commented Jun 9, 2022

xuezhou25 commented Jun 9, 2022

xuezhou25 commented Jun 9, 2022 • edited Loading

cliu123 commented Jun 10, 2022 • edited Loading

tlfeng commented Jun 10, 2022

cliu123 commented Jun 13, 2022 • edited Loading

reta commented Jun 9, 2022 •

edited

Loading

xuezhou25 commented Jun 9, 2022 •

edited

Loading

cliu123 commented Jun 10, 2022 •

edited

Loading

cliu123 commented Jun 13, 2022 •

edited

Loading