Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

Closed

Conversation

deshsidd
Copy link
Contributor

@deshsidd deshsidd commented Sep 27, 2023

  • Initial skeleton changes
  • Using QueryBuilderVisitor to visit all the nodes of the querybuilder tree
  • The tree referred to here is the tree of querybuilders for the incoming search query with the topLevelQueryBuilder as the root.
  • Integration with Metric Framework : Add Telemetry metrics framework #10241
  • Integration with QueryShapeVisitor to log the shape of the queries : https://gist.github.com/msfroh/74aa3fee52f4074c5e7b8d85f76e88ab
  • Need to also figure out all the types of queries we want to increment counters for

@deshsidd deshsidd changed the title Search Query Categorizor initial skeleton using QueryBuilderVisitor [Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor Sep 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 27, 2023

Compatibility status:

Checks if related components are compatible with change 3338d4f

Incompatible components

Skipped components

Compatible components

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@msfroh
Copy link
Collaborator

msfroh commented Oct 2, 2023

The sample implementation doesn't tell us whether given query clause is required or not, or if it's involved in scoring. These have major implications for performance.

@deshsidd deshsidd force-pushed the sid/search-query-categorizor branch from d1a1f2c to fadba4c Compare October 4, 2023 05:57
mch2 and others added 27 commits October 9, 2023 01:10
…ch-project#10347)

* Update multipart download path to write to temp files.

This change updates ReadContextListener to first write parts to a temp location
until all parts have been received.

Signed-off-by: Marc Handalian <[email protected]>

* Suppress forbidden IOUtils.fsync

Signed-off-by: Marc Handalian <[email protected]>

* Remove unnecessary logging format

Signed-off-by: Marc Handalian <[email protected]>

* sync directory after file rename

Signed-off-by: Marc Handalian <[email protected]>

* Remove flaky threadpool terminate test

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Marc Handalian <[email protected]>
…ject#10205)

Bumps org.apache.commons:commons-compress from 1.23.0 to 1.24.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…h-project#10016)

* fix stale remote cluster uuid state not purged from remote

Signed-off-by: bansvaru <[email protected]>

* fix tests

Signed-off-by: bansvaru <[email protected]>

* use new limit parameter

Signed-off-by: bansvaru <[email protected]>

* minor refactoring

Signed-off-by: bansvaru <[email protected]>

* delete index metadata files before manifest file

Signed-off-by: bansvaru <[email protected]>

* add basic UT

Signed-off-by: bansvaru <[email protected]>

* delete all data related to a cluster uuid in a single call

Signed-off-by: bansvaru <[email protected]>

* fix git diff

Signed-off-by: bansvaru <[email protected]>

* remove unreferenced code

Signed-off-by: bansvaru <[email protected]>

* fix spa

Signed-off-by: bansvaru <[email protected]>

---------

Signed-off-by: bansvaru <[email protected]>
…n/tools/plugin-cli (opensearch-project#10297)

* Bump org.bouncycastle:bc-fips in /distribution/tools/plugin-cli

Bumps org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4.

---
updated-dependencies:
- dependency-name: org.bouncycastle:bc-fips
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
* Race condition fix for datetime optimization

Signed-off-by: Prabhat Sharma <[email protected]>

* Changed JavaDateTimeFormatter caching of parser from MRU(most recently used) to a simple last used formatter

Signed-off-by: Prabhat Sharma <[email protected]>

---------

Signed-off-by: Prabhat Sharma <[email protected]>
Co-authored-by: Prabhat Sharma <[email protected]>
…ata fields in ingest simulate API (opensearch-project#10101)

* Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API

Signed-off-by: Gao Binlong <[email protected]>

* modify change log

Signed-off-by: Gao Binlong <[email protected]>

* Add more tests

Signed-off-by: Gao Binlong <[email protected]>

---------

Signed-off-by: Gao Binlong <[email protected]>
Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]>
Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>
…ases (opensearch-project#10370)

* Segment Replication - Fix ShardLockObtained error during corruption cases

This change fixes a bug where shards could not be recreated locally after corruption.
This occured because the store was not decref'd to 0 if the commit on close would fail
with a corruption exception.

Signed-off-by: Marc Handalian <[email protected]>

* Remove exra logs

Signed-off-by: Marc Handalian <[email protected]>

* Remove flaky assertion on store refcount

Signed-off-by: Marc Handalian <[email protected]>

* Remove flaky test.

Signed-off-by: Marc Handalian <[email protected]>

* PR Feedback.

Remove hacky handling of corruption when fetching metadata.  This will now check for store corruption
when replication has failed and fail the shard accordingly.

This commit also fixes logging in NRTReplicationEngine.

Signed-off-by: Marc Handalian <[email protected]>

* Fix unit test.

Signed-off-by: Marc Handalian <[email protected]>

* Fix test failure testSegRepSucceedsOnPreviousCopiedFiles.

This test broke because we invoked target.indexShard on a closed replicationTarget.
In these cases we can assume the store is not corrupt.

Signed-off-by: Marc Handalian <[email protected]>

* spotless

Signed-off-by: Marc Handalian <[email protected]>

* Revert flaky IT

Signed-off-by: Marc Handalian <[email protected]>

* Fix flakiness failure by expecting RTE when check index fails.

Signed-off-by: Marc Handalian <[email protected]>

* reintroduce ITs and use recoveries API instead of waiting on shard state.

Signed-off-by: Marc Handalian <[email protected]>

* Fix edge case where flush failures would not get reported as corruption.

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Marc Handalian <[email protected]>
…arch-project#10402)

* Fix stats reporting for multistream downloads.

Signed-off-by: Rishikesh1159 <[email protected]>

* rename tracker to fileTransferTracker.

Signed-off-by: Rishikesh1159 <[email protected]>

---------

Signed-off-by: Rishikesh1159 <[email protected]>
… allowed in base64UUID url charset (opensearch-project#10379)

* Refactor Remote Store Metadata Lock Manager Utils

Signed-off-by: Harish Bhakuni <[email protected]>

* Address PR Comments

Signed-off-by: Harish Bhakuni <[email protected]>

* Address PR Comments

Signed-off-by: Harish Bhakuni <[email protected]>

* Update Changelog entry

Signed-off-by: Harish Bhakuni <[email protected]>

* Update Changelog entry

Signed-off-by: Harish Bhakuni <[email protected]>

* Unmute testDeleteShallowCopySnapshot test

Signed-off-by: Harish Bhakuni <[email protected]>

---------

Signed-off-by: Harish Bhakuni <[email protected]>
Co-authored-by: Harish Bhakuni <[email protected]>
…25.1 in /plugins/repository-gcs (opensearch-project#10298)

* Bump com.google.api.grpc:proto-google-common-protos

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1.
- [Release notes](https://github.com/googleapis/sdk-platform-java/releases)
- [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/sdk-platform-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…project#10437)

* DiversifiedSamplerIT.testNestedSamples
* QueryProfilePhaseTests.testMaxScore
* QueryProfilePhaseTests.testCollapseQuerySearchResults
* HighlighterSearchIT.testHighlightQueryRewriteDatesWithNow
* FieldCapabilitiesIT.testWithIndexFilter
* QueryProfilePhaseTests.testDisableTopScoreCollection

Signed-off-by: Jay Deng <[email protected]>
Signed-off-by: Jay Deng <[email protected]>
…25.1 in /plugins/repository-gcs (opensearch-project#10298) (opensearch-project#10461)

* Bump com.google.api.grpc:proto-google-common-protos

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1.
- [Release notes](https://github.com/googleapis/sdk-platform-java/releases)
- [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/sdk-platform-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-type: direct:production
  update-type: version-update:semver-patch
...



* Updating SHAs



* Update changelog



---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Signed-off-by: Siddhant Deshmukh <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.