Akka to Pekko Migration Rollback

[nmalonzo]

Hi,

I am exploring what approach to use in migrating production code from Akka to Pekko. From a cursory checking it seems to just require replacing the name of several configs to pekko versions. However, when I look on rolling back there seems to be at least one issue.. This is a deal-breaker for me for doing a simple rolling deployment from the akka project to the pekko project. I am leaning on designing a more complicated, but hopefully more robust, method using migration classes that will do live migration between an akka cluster and its pekko cluster version.

This will have a number of challenges, primarily allowing intra-cluster communication between state nodes. Is there anyone in the community having migrated production code to pekko and needing to rollback for some reason?

[pjfanning]

@nmalonzo would you be able to provide a more detailed breakdown of what you think is missing in Pekko - that blocks your migration? Maybe you could share the design you have in mind for a more 'robust' solution but it would be nice to spell out the problem(s) that it solves.

#837 is merged - is there a specific reason that you are unhappy with it?

Pekko is a volunteer project and the work to support Akka/Pekko interop has been done using small scale scenarios. We haven't really and anyone come forward who has a real world application running that needs this support. It wouldn't surprise me if there are gaps.

I look forward to seeing what you submit and hopefully others will see this discussion and help out.

As well as setting up the configs by copying the existing configs for your Akka cluster and changing the names to start with 'pekko' - you should read https://cwiki.apache.org/confluence/display/PEKKO/Pekko+Akka+Compatibility

[mdedetrich]

Thanks again for the reply. My main question with Akka Pekko compatibility is will it work and be stable indefinitely?

My main question is can Akka and Pekko actors coexist on the same cluster indefinitely?

We would have to discuss this as it would be quite a high maintenance burden to actually guarantee this. The original intent of this migration capability is for people to do a gradual rollover migration to a Pekko cluster which implies shutting down the Akka nodes.

Having to maintain both Akka and Pekko nodes indefinitely was not part of that original intention and historically we have had the issue where while we implemented these migration features, we didn't garner a lot of feedback about said features. Ontop of this, having to maintain such compatibility means we cannot introduce changes that may break Akka nodes in possible future releases of Pekko even though aside from migration, Akka is not something explicitly support.

[nmalonzo]

Thanks. Pardon me if I clarify some things. The wiki says join Pekko nodes to an Akka cluster. Do you mean starting Pekko nodes while Akka nodes are still alive? The Akka node cluster coordinator will detect these pekko nodes and treat them as akka nodes? When all akka nodes are terminated the cluster coordinator will be transferred to a pekko node pretending to be an akka node. At this point it is still an Akka cluster?

Once all Akka pretending pekko nodes have been configured as Pekko nodes then it will be a Pekko cluster? So while rolling deployment is happening to change these nodes one by one to Pekko nodes presenting as Pekko nodes, two clusters will exist? An akka cluster and a pekko cluster?

A persistent entity on a node migrated to Pekko is not guaranteed to be able to be rolled back to Akka right? A pekko node pretending to be an Akka node can change the journal/snapshot such that it cannot be loaded to an Akka node? So coupled with the lack of guarantee in Akka Pekko persistence (even with a freezed Pekko version) the safe way to migrate from Akka to Pekko is to maintain a separate Akka and Pekko cluster while migration is ongoing. The problem with this is we have to deploy a GRPC endpoint to the persistent state nodes to be able to communicate with the other cluster. But with no guarantee of roll back to Akka can be done even with Akka Pekko coexistence it is the only choice.

[pjfanning]

All I can keep doing is repeating the mantra that users should trial out the migration in a test cluster. I have no access to your set up to advise on specifics.

The only specific issue that I am aware of is that #837 will cause issues for Akka nodes in the cluster because any Pekko node added to the cluster will attempt to persist the any snapshots that it is asked to handle with a modified serializer name. That is what #1423 attempts to provide a solution for.

I will also repeat that Apache Pekko is free software with no warranties provided.

I don't know how complicated your Akka set up is but it may that Apache Pekko may not be a good fit for you.

[pjfanning]

The wiki page describes migrating a cluster from a pure Akka cluster to a pure Pekko cluster involves 3 distinct phases.

[...] you will need to do 3 phases to fully replace your Akka cluster with a Pekko cluster:

join Pekko nodes (that impersonate Akka nodes) to your Akka cluster and then start to remove the Akka nodes
when you have no Akka nodes, start replacing the Pekko nodes with ones that do not impersonate Akka nodes but that still agree to interact with Akka nodes (because some Pekko nodes are still impersonating Akka nodes - ie ones that still have pekko.remote.protocol-name="akka")
when all nodes are using pekko.remote.protocol-name="pekko", then you can replace all the nodes with nodes that no longer accept "pekko" as a protocol name (ie pekko.remote.accept-protocol-names=["pekko"])

If you hit issues in the first phase, then you can stop the new Pekko nodes and return to being a cluster with just Akka nodes.

[mdedetrich]

A persistent entity on a node migrated to Pekko is not guaranteed to be able to be rolled back to Akka right? A pekko node pretending to be an Akka node can change the journal/snapshot such that it cannot be loaded to an Akka node?

This is why Pekko 1.0.x version series exists, its guaranteed not to have any behavior changes compared to Akka 2.6 unless we find critical bugs/security issues. The idea is that people migrate their current Akka cluster to Pekko 1.0.x, make sure all of the Akka nodes are shut down and then they can start migrating their new Pekko 1.0.x cluster to a later version of Pekko (i.e. 1.1.x).

As long as you stick with Pekko 1.0.x, you can theoretically move back to Akka (for whatever reason) but the idea is that you eventually move away from Pekko 1.0.x. There is no realistic expectation that Pekko versions after 1.0.x will continue to work with Akka nodes, and its not just for technical reasons. You should view Pekko as separate open source software, with 1.0.x being a special case designed to aid in migration.

[nmalonzo]

Thanks for the reply. It seems even if we do a lot of testing, migrating a production environment from Akka 2.6 to Pekko 1.0.X and for some reason rolling it back to Akka 2.6 will entail risk in the persistence tables (journal and snapshot). This is why I am now favoring starting a separate Pekko cluster and doing a live migration from the Akka cluster to the Pekko cluster while both clusters are up. The problem with this though is it would be much harder for nodes between two clusters to communicate. I am thinking of putting a GRPC endpoint in the persistent state nodes to allow migration between them. Would you know an easier way to do this or am I going in the right direction? Our requirement is that there should be virtually zero risk of corrupting the persistent actor data during migration. If there is a small risk that migrating the persistent akka actors to pekko 1.0.X and rolling them back to akka will corrupt their data, we cannot use it.

…

On Tue, Aug 6, 2024 at 10:05 PM Matthew de Detrich ***@***.***> wrote: A persistent entity on a node migrated to Pekko is not guaranteed to be able to be rolled back to Akka right? A pekko node pretending to be an Akka node can change the journal/snapshot such that it cannot be loaded to an Akka node? This is why Pekko 1.0.x version series exists, its guaranteed not to have any behavior changes compared to Akka 2.6 unless we find critical bugs/security issues. The idea is that people migrate their current Akka cluster to Pekko 1.0.x, make sure all of the Akka nodes are shut down and then they can start migrating their new Pekko 1.0.x cluster to a later version of Pekko (i.e. 1.1.x). As long as you stick with Pekko 1.0.x, you can theoretically move back to Akka (for whatever reason) but the idea is that you eventually move away from Pekko 1.0.x. There is however no realistic expectation that Pekko versions after 1.0.x will continue to work with Akka nodes, and its not just for technical reasons. You should view Pekko as separate open source software, with 1.0.x being a special case designed to aid in migration. — Reply to this email directly, view it on GitHub <#1421 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF6CF54DYLLDK4UGOZ6HPSTZQDJ3BAVCNFSM6AAAAABL2CPBD6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRVGQZTIMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>