[BUG] org.opensearch.search.aggregations.bucket.DateRangeIT.testPartiallyUnmapped {p0={"search.concurrent_segment_search.enabled":"false"}}

[sohami]

Describe the bug
The test case org.opensearch.search.aggregations.bucket.DateRangeIT.testPartiallyUnmapped {p0={"search.concurrent_segment_search.enabled":"false"}} is flaky
See: https://build.ci.opensearch.org/job/gradle-check/27311/testReport/junit/org.opensearch.search.aggregations.bucket/DateRangeIT/testPartiallyUnmapped__p0___search_concurrent_segment_search_enabled___false___/

java.lang.AssertionError: Unexpected ShardFailures: [shard [[oFevbBxtQsy8MR4fkfKdlw][idx][1]], reason [RemoteTransportException[[node_s1][127.0.0.1:45191][indices:data/read/search[phase/dfs]]]; nested: OpenSearchParseException[failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]: [failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]; ], cause [OpenSearchParseException[failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]: [failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];
	at org.opensearch.common.time.JavaDateMathParser.parseDateTime(JavaDateMathParser.java:247)
	at org.opensearch.common.time.JavaDateMathParser.parse(JavaDateMathParser.java:89)
	at org.opensearch.search.DocValueFormat$DateTime.parseLong(DocValueFormat.java:299)
	at org.opensearch.search.DocValueFormat$DateTime.parseDouble(DocValueFormat.java:304)
	at org.opensearch.search.aggregations.bucket.range.DateRangeAggregationBuilder.lambda$innerBuild$2(DateRangeAggregationBuilder.java:351)
	at org.opensearch.search.aggregations.bucket.range.AbstractRangeBuilder.processRanges(AbstractRangeBuilder.java:104)
	at org.opensearch.search.aggregations.bucket.range.DateRangeAggregationBuilder.innerBuild(DateRangeAggregationBuilder.java:334)
	at org.opensearch.search.aggregations.bucket.range.DateRangeAggregationBuilder.innerBuild(DateRangeAggregationBuilder.java:57)
	at org.opensearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:411)
	at org.opensearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:60)
	at org.opensearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:155)
	at org.opensearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:459)
	at org.opensearch.search.SearchService.parseSource(SearchService.java:1294)
	at org.opensearch.search.SearchService.createContext(SearchService.java:998)
	at org.opensearch.search.SearchService.executeDfsPhase(SearchService.java:526)
	at org.opensearch.search.SearchService$1.lambda$onResponse$0(SearchService.java:511)
	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.IllegalArgumentException: failed to parse date field [1329264000000] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]
	at org.opensearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:268)
	at org.opensearch.common.time.JavaDateMathParser.parseDateTime(JavaDateMathParser.java:238)
	... 26 more
Caused by: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers]
	at org.opensearch.common.time.JavaDateFormatter.doParse(JavaDateFormatter.java:307)
	at org.opensearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:266)
	... 27 more
]]
Expected: <0>
     but: was <1>

To Reproduce
Don't see a command in the failed CI

Expected behavior
Test should pass all the time

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):
N/A

Additional context
N/A

[reta]

CC @CaptainDredge

[CaptainDredge]

@reta can you please close this? Not able to repro it

[dblock]

@CaptainDredge It clearly failed when this issue was opened, that's why it's flaky. It mostly passes, except when otherwise. No repro is not a good reason to close ;)

[reta]

@CaptainDredge It clearly failed when this issue was opened, that's why it's flaky. It mostly passes, except when otherwise. No repro is not a good reason to close ;)

@dblock that is correct, to reconstruct the picture:

• Added performance improvement for datetime field parsing #9567 was merged and this was the change causing the issue
• Race condition fix for datetime optimization #10385 was merged in a day (or two) after fixing the issue

There was some timeframe the tests were failing and the fact issue is not reproducible anymore is a good sign (still watching it).