Prerequisite: Have an environment with just with 2 worker nodes or taint 1 out of 3 worker node to be NoExecute & NoSchedule.
+This will serve as a constrained fallback and limited source of recovery in the event of failure.
1. Kill the engines and instance manager repeatedly
+Given 1 RWO and 1 RWX volume is attached to a pod.
+And Both the volumes have 2 replicas.
+And Random data is continuously being written to the volume using command dd if=/dev/urandom of=file1 count=100 bs=1M conv=fsync status=progress oflag=direct,sync
When One replica rebuilding is triggered by crashing the IM +AND Immediately IM associated with another replica is crashed +AND After crashing IMs, detaching of Volume is tried either by pod deletion or using Longhorn UI
+Then Volume should not stuck in attaching-detaching loop
+When Volume is detached and manually attached again. +And Engine running on the node where is volume is attached in killed
+Then Volume should recover once the engine is back online.
+2. Illegal values in Volume/Snap.meta
+Given 1 RWO and 1 RWX volume is attached to a pod. +And Both the volumes have 2 replicas.
+When Some random values are set in the Volume/snap meta file +And If replica rebuilding is triggered and the IM associated with another replica is also crashed
+Then Volume should not stuck in attaching-detaching loop
+3. Deletion of Volume/Snap.meta
+Given 1 RWO and 1 RWX volume is attached to a pod. +And Both the volumes have 2 replicas.
+When The Volume & snap meta files are deleted one by one. +And If replica rebuilding is triggered and the IM associated with another replica is also crashed
+Then Volume should not stuck in attaching-detaching loop
+4. Failed replica tries to rebuild from other just crashed replica - https://github.com/longhorn/longhorn/issues/4212
+Given 1 RWO and 1 RWX volume is attached to a pod.
+And Both the volumes have 2 replicas.
+And Random data is continuously being written to the volume using command dd if=/dev/urandom of=file1 count=100 bs=1M conv=fsync status=progress oflag=direct,sync
When One replica rebuilding is triggered by crashing the IM +AND Immediately IM associated with another replica is crashed
+Then Volume should not stuck in attaching-detaching loop.
+5. Volume attachment Modification/deletion
+Given A deployment and statefulSet are created with same name and attached to Longhorn Volume. +AND Some data is written and their md5sum is computed
+When The statefulSet and Deployment are deleted without deleting the volumes +And Same named new statefulSet and Deployment are created with new PVCs. +And Before above deployed workload could attach to volumes, attached node is rebooted
+Then After node reboot completion, volumes should reflect right status. +And the newly created deployment and statefulSet should get attached to the volumes.
+When The volume attachments of above workloads are deleted. +And above workloads are deleted and recreated immediately.
+Then No multi attach or other errors should be observed.
+6. Use monitoring/word press/db workloads
+Given Monitoring and word press and any other db related workload are deployed in the system
+And All the volumes have 2 replicas.
+And Random data is continuously being written to the volume using command dd if=/dev/urandom of=file1 count=100 bs=1M conv=fsync status=progress oflag=direct,sync
When One replica rebuilding is triggered by crashing the IM +AND Immediately IM associated with another replica is crashed
+Then Volume should not stuck in attaching-detaching loop.
+ +