Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate metadata log entry data model and persistence #406

Merged
merged 10 commits into from
Jul 15, 2024
Merged

Separate metadata log entry data model and persistence #406

merged 10 commits into from
Jul 15, 2024

Conversation

seankao-az
Copy link
Collaborator

@seankao-az seankao-az commented Jun 30, 2024 •
edited
Loading

Description

  • Refactor FlintMetadataLogEntry to be generic to data storage
  • Remove dataSourceName from DefaultOptimisticTransaction
  • Move persistence logic for OpenSearch away from FlintMetadataLogEntry and DefaultOptimisticTransaction into FlintMetadataLogEntryOpenSearchConverter

Before

FlintMetadataLogEntry(
    id: String,
    seqNo: Long,
    primaryTerm: Long,
    createTime: Long,
    state: IndexState,
    dataSource: String,
    error: String)

After

FlintMetadataLogEntry(
    id: String,
    createTime: Long,
    state: IndexState,
    entryVersion: Map[String, Any],
    error: String,
    properties: Map[String, Any])
  • entryVersion is used for FlintSpark updateIndex. For OpenSearch, it should specify seqNo and primaryTerm
  • properties can be used to store additional context required for persistence. For example, dataSourceName for OpenSearch

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@seankao-az
update metadata log entry model to be generic
9712905 
Signed-off-by: Sean Kao <[email protected]>
@seankao-az
move OS specific stuff away from log entry
0cb458b 
Signed-off-by: Sean Kao <[email protected]>

refactor: utils for OS log entry storage

Signed-off-by: Sean Kao <[email protected]>
@seankao-az
remove index name from log entry
d0adefc 
Signed-off-by: Sean Kao <[email protected]>
@seankao-az
add log entry storage context
37727fe 
Signed-off-by: Sean Kao <[email protected]>
@seankao-az seankao-az mentioned this pull request Jul 1, 2024
Closed
@seankao-az seankao-az changed the title [Refactor] Generalize log entry storage Generalize log entry storage Jul 2, 2024
@seankao-az seankao-az self-assigned this Jul 2, 2024
@seankao-az seankao-az added maintenance Code refactoring 0.5 labels Jul 2, 2024
@seankao-az seankao-az changed the title Generalize log entry storage Generalize metadata log entry storage Jul 2, 2024
@seankao-az seankao-az changed the title Generalize metadata log entry storage Separate metadata log entry data model and persistence Jul 2, 2024
@seankao-az seankao-az marked this pull request as ready for review July 2, 2024 07:21
@seankao-az seankao-az force-pushed the log-entry-storage branch from d0e8f33 to 79cf456 Compare July 2, 2024 18:16
dai-chen
dai-chen approved these changes Jul 15, 2024
View reviewed changes
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

@seankao-az seankao-az merged commit 978c028 into opensearch-project:main Jul 15, 2024
4 checks passed
@seankao-az seankao-az mentioned this pull request Jul 15, 2024
Closed
seankao-az added a commit to seankao-az/opensearch-spark that referenced this pull request Aug 9, 2024
@seankao-az
Separate metadata log entry data model and persistence (opensearch-pr…
c3bbe63 
…oject#406)

* update metadata log entry model to be generic

Signed-off-by: Sean Kao <[email protected]>

* move OS specific stuff away from log entry

Signed-off-by: Sean Kao <[email protected]>

refactor: utils for OS log entry storage

Signed-off-by: Sean Kao <[email protected]>

* remove index name from log entry

Signed-off-by: Sean Kao <[email protected]>

* add log entry storage context

Signed-off-by: Sean Kao <[email protected]>

* fix error

Signed-off-by: Sean Kao <[email protected]>

* rename storage utils to OS converter and add test

Signed-off-by: Sean Kao <[email protected]>

* update comment for log entry for OpenSearch

Signed-off-by: Sean Kao <[email protected]>

* rename storageContext to properties

Signed-off-by: Sean Kao <[email protected]>

* remove unused failLogEntry

Signed-off-by: Sean Kao <[email protected]>

* use jackson log entry toJson

Signed-off-by: Sean Kao <[email protected]>

---------

Signed-off-by: Sean Kao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dai-chen dai-chen dai-chen approved these changes

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@kaituo kaituo Awaiting requested review from kaituo kaituo is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

Assignees

@seankao-az seankao-az

Labels
0.5 maintenance Code refactoring
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@seankao-az @dai-chen