feat(lep): auto-salvage support for v2 volumes

[c3y1huang]

Which issue(s) this PR fixes:

Issue #8430

What this PR does / why we need it:

Proposes extending Longhorn's auto-salvage feature to support v2 volumes

Special notes for your reviewer:

None

Additional documentation or context

https://suse.slack.com/archives/C02DR3N5T24/p1713926457976349

[DamiaSan]

This functionality has some overlaps with the delta rebuilding, for example in the case when the node where the raid resides restart (we must elect an healty replica and compare all others replica with this one to ensure they are aligned). So I think we need an unique implementation for these 2 functionalities. WDYT?
cc @innobead

[c3y1huang]

This functionality has some overlaps with the delta rebuilding, for example in the case when the node where the raid resides restart (we must elect an healty replica and compare all others replica with this one to ensure they are aligned). So I think we need an unique implementation for these 2 functionalities. WDYT?

Based on the Slack discussion: There is no way to guarantee that there won't be any data block missing in the head when the volume suddenly crashes. As @shuo-wu suggested, our best approach, for now, is to ensure that the replica candidates have the complete snapshot set. I don't see the overlap between this proposal and the future integration with the delta rebuilding as this proposal leans more towards the control plan selection of the reusable candidate. Could you elaborate on the potential overlap?

IMO we can enhance the handling of the scenario when the RAID node goes offline with delta rebuilding. An approach is to address this in the engine instance, similar to how v1 volume handles it, or incorporate additional information in Replica.Status.Snapshots? What do you think?

@derekbit @shuo-wu , any thoughts about this?

[DamiaSan]

This functionality has some overlaps with the delta rebuilding, for example in the case when the node where the raid resides restart (we must elect an healty replica and compare all others replica with this one to ensure they are aligned). So I think we need an unique implementation for these 2 functionalities. WDYT?

Based on the Slack discussion: There is no way to guarantee that there won't be any data block missing in the head when the volume suddenly crashes. As @shuo-wu suggested, our best approach, for now, is to ensure that the replica candidates have the complete snapshot set. I don't see the overlap between this proposal and the future integration with the delta rebuilding as this proposal leans more towards the control plan selection of the reusable candidate. Could you elaborate on the potential overlap?

IMO we can enhance the handling of the scenario when the RAID node goes offline with delta rebuilding. An approach is to address this in the engine instance, similar to how v1 volume handles it, or incorporate additional information in Replica.Status.Snapshots? What do you think?

@derekbit @shuo-wu , any thoughts about this?

The overlap is the selection method and the check that replicas have the complete snapshot set.

[c3y1huang]

Summary from the LEP meeting:

1. Remove Replica.Status.Snapshots from this proposal. There shouldn't be a scenario for inconsistent snapshot sets between failed reusable replicas.
2. Filter replica candidates based on the size of the head volume.
3. Block trim operations when the volume is degraded to maintain reliable volume head size.

[c3y1huang]

The CI codespell check failed to build. I'm not sure if it's a temporary thing. It's working again.

[derekbit]

LGTM. The LEP is quite straightforward.