Blob-store based oplog archive #538
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
justcoon
reviewed
May 30, 2024
| @@ -155,9 +331,14 @@ pub enum BlobStorageNamespace { | |||
| account_id: AccountId, | |||
| worker_id: WorkerId, | |||
| }, | |||
| CompressedOplog { | |||
| account_id: AccountId, | |||
| component_id: ComponentId, | |||
There was a problem hiding this comment.
is there reason why here is component_id and not worker_id?
There was a problem hiding this comment.
This way it was easy to implement the equivalent of redis SCAN without changing / extending the blob storage interface.
afsalthaj
approved these changes
May 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #216
Contains refactorings related to #502
Introduces
OwnedWorkerId, a combination ofWorkerIdandAccountIdand uses it in most places in the worker executor where previously worker and account IDs were passed separately. Also the number of places where this type is needed (the additional account ID) increased because now the oplog requires to have an associated account ID for each operation. Although this change increases the required information to be transferred in many cases, it unified the API a bit as now everything requires a combination of worker id and account id: this can be explained by worker executor needs to know who owns the worker, but it does not want to query an external service to determine so - so this information must be passed from the outer layers where authorization already calculated the information anyway (or in case all worker belongs to the same account, it is passed as a constant value).The primary reason for passing this ownership information down to the oplog service is to be able to store oplog archives grouped by accounts, similarly how custom blobs are already stored grouped by accounts. This is good for various security and privacy reasons and can support some planned features.