Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve support for elasticsearch replication #66

Merged
merged 7 commits into from
Aug 30, 2024

Conversation

eminano
Copy link
Collaborator

@eminano eminano commented Aug 29, 2024

This PR improves the support for elasticsearch replication, as well as improving the feature support for opensearch replication. Before this PR we were using a go-elasticsearch library version that supported opensearch (before it introduced version/client/server checks in 2021). This prevented any updates, and it had limitations when using vector fields with elasticsearch.

By separating the opensearch and elasticsearch clients we can address the slight differences for each in the index, mapping and overall library use. The separation is done at the library/pkg level, the rest of changes are meant to align with a decoupled approach, avoiding leaking implementation details.

Main changes:

  • Addition of elasticsearch client and creation of an abstracted search store library with the shared functionality, used as an access point for the rest of the code, abstracting the implementation details.
  • Addition of a search store mapper, along with the opensearch and elasticsearch implementations, that takes care of the specificities of each regarding the mapping (ex., knn_vector vs dense_vector) .
  • Update search batch processor to use the newly abstracted library
  • Update configuration to be able to provide an elasticsearch or opensearch store url
  • Addition of integration tests to cover the postgres to elasticsearch integration

The commits have been split for ease of reviewing.

@eminano eminano requested a review from exekias August 29, 2024 15:23
Copy link
Member

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM besides user input handling!

@eminano eminano merged commit bab0a8e into main Aug 30, 2024
5 checks passed
@eminano eminano deleted the add-full-support-for-elasticsearch branch August 30, 2024 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants