Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Khepri post-migration step to not delete certain files or dirs #11304

Open
gomoripeti opened this issue May 22, 2024 · 1 comment
Open

Comments

@gomoripeti
Copy link
Contributor

Is your feature request related to a problem? Please describe.

As the last step of migrating to Khepri (ie enabling the khepri_db feature flag) all Mnesia related files are deleted (https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_khepri.erl#L1469-L1480) However the heuristic to find Mnesia related files is too eager, it deletes everything in the data dir recursively except for those which are known to be not Mnesia (ie msg_stores, coordination, quorum, stream and rabbitmq_metadata dirs are not deleted).

This is an issue for community plugins which want to migrate away from Mnesia into a new subdir. (They might not even use Khepri, or use Khepri but a different store id.) As an example (and the motivation of this ticket) the delayed message store is planned to store its leveldb data in a dmx subdir (https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/pull/253/files#diff-64860364dca47a85f9e9fc932a30c53a07673f78e50ebc2a622e64bf78e91ce2R41-R42). After the delayed message plugin nicely migrated its messages to a new storage backend, the post-migration step wipes the newly created dmx dir. There is no clean callback that is called after this step that the plugin can implement to eg move the migrated data from a tmp dir to its final location.

Describe the solution you'd like

I don't have a strong idea for a solution. One option is that plugins can register a list of additional files and dirs (maybe an optional callback for the migration behaviour) that are not deleted in the post-migration step.

Describe alternatives you've considered

Alternative could be an optional migration callback that is executed after rabbit_khepri:khepri_db_migration_post_enable/0 that the plugins can implement to move the migrated data from a tmp dir to its final location.

A workaround could be that the community plugin stores its data under one of the dirs which are not deleted, but this is a hack. The plugin data does not belong there.

Additional context

No response

@michaelklishin
Copy link
Member

A set of patterns under khepri.migration.delete_protected_path_patterns should do. Together with a way to exclude certain patterns in https://github.com/rabbitmq/khepri_mnesia_migration, in all likelihood.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants