Skip to content

Commit

Permalink
refactor(negative): improve test readabiltity
Browse files Browse the repository at this point in the history 
longhorn/longhorn-7034

Signed-off-by: Chin-Ya Huang <[email protected]>
  • Loading branch information
@c3y1huang
c3y1huang committed Mar 6, 2024
1 parent 0fe08e4 commit 724833e
Show file tree
Hide file tree
Showing 6 changed files with 230 additions and 155 deletions.
39 changes: 26 additions & 13 deletions e2e/tests/cluster_restart.robot
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
*** Settings ***
Documentation Negative Test Cases
Resource ../keywords/common.resource
Resource ../keywords/deployment.resource
Resource ../keywords/node.resource
Resource ../keywords/longhorn.resource
Resource ../keywords/host.resource
Resource ../keywords/persistentvolumeclaim.resource
Resource ../keywords/statefulset.resource
Resource ../keywords/workload.resource

Expand All @@ -17,22 +20,32 @@ ${CONTROL_PLANE_NODE_NETWORK_LATENCY_IN_MS} 0

*** Test Cases ***
Restart Cluster While Workload Heavy Writing
Given Create deployment 0 with rwo volume
And Create deployment 1 with rwx volume
And Create deployment 2 with rwo and strict-local volume
And Create statefulset 0 with rwo volume
And Create statefulset 1 with rwx volume
And Create statefulset 2 with rwo and strict-local volume
Given Create persistentvolumeclaim 0 using RWO volume
And Create persistentvolumeclaim 1 using RWX volume
And Create persistentvolumeclaim 2 using RWO volume with strict-local storageclass
And Create deployment 0 with persistentvolumeclaim 0
And Create deployment 1 with persistentvolumeclaim 1
And Create deployment 2 with persistentvolumeclaim 2
And Create statefulset 0 using RWO volume
And Create statefulset 1 using RWX volume
And Create statefulset 2 using RWO volume with strict-local storageclass

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to deployment 0
And Keep writing data to deployment 1
And Keep writing data to deployment 2
And Keep writing data to statefulset 0
And Keep writing data to statefulset 1
And Keep writing data to statefulset 2
And Keep writing data to pod of deployment 0
And Keep writing data to pod of deployment 1
And Keep writing data to pod of deployment 2
And Keep writing data to pod of statefulset 0
And Keep writing data to pod of statefulset 1
And Keep writing data to pod of statefulset 2

When Restart cluster
And Wait for longhorn ready
And Wait for deployment 0 pods stable
And Wait for deployment 1 pods stable
And Wait for deployment 2 pods stable
And Wait for statefulset 0 pods stable
And Wait for statefulset 1 pods stable
And Wait for statefulset 2 pods stable

Then Check deployment 0 works
And Check deployment 1 works
Expand Down
20 changes: 14 additions & 6 deletions e2e/tests/kubelet_restart.robot
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
*** Settings ***
Documentation Negative Test Cases
Resource ../keywords/common.resource
Resource ../keywords/persistentvolumeclaim.resource
Resource ../keywords/statefulset.resource
Resource ../keywords/workload.resource

Expand All @@ -14,21 +16,27 @@ ${RETRY_INTERVAL} 1

*** Test Cases ***
Restart Volume Node Kubelet While Workload Heavy Writing
Given Create statefulset 0 with rwo volume
Given Create statefulset 0 using RWO volume

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to statefulset 0
And Keep writing data to pod of statefulset 0

When Stop volume node kubelet of statefulset 0 for 10 seconds
And Wait for volume of statefulset 0 healthy
And Wait for statefulset 0 stable
And Wait for statefulset 0 pods stable

Then Check statefulset 0 works
END

Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing
Given Create statefulset 0 with rwo volume
Given Create statefulset 0 using RWO volume

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to statefulset 0
And Keep writing data to pod of statefulset 0

When Stop volume node kubelet of statefulset 0 for 360 seconds
And Wait for volume of statefulset 0 healthy
And Wait for statefulset 0 stable
And Wait for statefulset 0 pods stable

Then Check statefulset 0 works
END
165 changes: 108 additions & 57 deletions e2e/tests/node_reboot.robot
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
*** Settings ***
Documentation Negative Test Cases
Resource ../keywords/common.resource
Resource ../keywords/deployment.resource
Resource ../keywords/node.resource
Resource ../keywords/longhorn.resource
Resource ../keywords/host.resource
Resource ../keywords/persistentvolumeclaim.resource
Resource ../keywords/recurringjob.resource
Resource ../keywords/statefulset.resource
Resource ../keywords/volume.resource
Expand All @@ -15,30 +18,41 @@ Test Teardown Cleanup test resources
${LOOP_COUNT} 1
${RETRY_COUNT} 300
${RETRY_INTERVAL} 1
${VOLUME_TYPE} rwo
${VOLUME_TYPE} RWO
${CONTROL_PLANE_NODE_NETWORK_LATENCY_IN_MS} 0

*** Test Cases ***
Reboot Node One By One While Workload Heavy Writing
Given Create deployment 0 with rwo volume
And Create deployment 1 with rwx volume
And Create deployment 2 with rwo and strict-local volume
And Create statefulset 0 with rwo volume
And Create statefulset 1 with rwx volume
And Create statefulset 2 with rwo and strict-local volume
Given Create persistentvolumeclaim 0 using RWO volume
And Create persistentvolumeclaim 1 using RWX volume
And Create persistentvolumeclaim 2 using RWO volume with strict-local storageclass

And Create deployment 0 with persistentvolumeclaim 0
And Create deployment 1 with persistentvolumeclaim 1
And Create deployment 2 with persistentvolumeclaim 2

And Create statefulset 0 using RWO volume
And Create statefulset 1 using RWX volume
And Create statefulset 2 using RWO volume with strict-local storageclass

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to deployment 0
And Keep writing data to deployment 1
And Keep writing data to deployment 2
And Keep writing data to statefulset 0
And Keep writing data to statefulset 1
And Keep writing data to statefulset 2
And Keep writing data to pod of deployment 0
And Keep writing data to pod of deployment 1
And Keep writing data to pod of deployment 2
And Keep writing data to pod of statefulset 0
And Keep writing data to pod of statefulset 1
And Keep writing data to pod of statefulset 2

When Reboot node 0
And Reboot node 1
And Reboot node 2
And Wait for longhorn ready
And Wait for deployment 0 pods stable
And Wait for deployment 1 pods stable
And Wait for deployment 2 pods stable
And Wait for statefulset 0 pods stable
And Wait for statefulset 1 pods stable
And Wait for statefulset 2 pods stable

Then Check deployment 0 works
And Check deployment 1 works
Expand All @@ -49,25 +63,36 @@ Reboot Node One By One While Workload Heavy Writing
END

Power Off Node One By Once For More Than Pod Eviction Timeout While Workload Heavy Writing
Given Create deployment 0 with rwo volume
And Create deployment 1 with rwx volume
And Create deployment 2 with rwo and strict-local volume
And Create statefulset 0 with rwo volume
And Create statefulset 1 with rwx volume
And Create statefulset 2 with rwo and strict-local volume
Given Create persistentvolumeclaim 0 using RWO volume
And Create persistentvolumeclaim 1 using RWX volume
And Create persistentvolumeclaim 2 using RWO volume with strict-local storageclass

And Create deployment 0 with persistentvolumeclaim 0
And Create deployment 1 with persistentvolumeclaim 1
And Create deployment 2 with persistentvolumeclaim 2

And Create statefulset 0 using RWO volume
And Create statefulset 1 using RWX volume
And Create statefulset 2 using RWO volume with strict-local storageclass

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to deployment 0
And Keep writing data to deployment 1
And Keep writing data to deployment 2
And Keep writing data to statefulset 0
And Keep writing data to statefulset 1
And Keep writing data to statefulset 2
And Keep writing data to pod of deployment 0
And Keep writing data to pod of deployment 1
And Keep writing data to pod of deployment 2
And Keep writing data to pod of statefulset 0
And Keep writing data to pod of statefulset 1
And Keep writing data to pod of statefulset 2

When Power off node 0 for 6 mins
And Power off node 1 for 6 mins
And Power off node 2 for 6 mins
And Wait for longhorn ready
And Wait for deployment 0 pods stable
And Wait for deployment 1 pods stable
And Wait for deployment 2 pods stable
And Wait for statefulset 0 pods stable
And Wait for statefulset 1 pods stable
And Wait for statefulset 2 pods stable

Then Check deployment 0 works
And Check deployment 1 works
Expand All @@ -78,23 +103,34 @@ Power Off Node One By Once For More Than Pod Eviction Timeout While Workload Hea
END

Reboot All Worker Nodes While Workload Heavy Writing
Given Create deployment 0 with rwo volume
And Create deployment 1 with rwx volume
And Create deployment 2 with rwo and strict-local volume
And Create statefulset 0 with rwo volume
And Create statefulset 1 with rwx volume
And Create statefulset 2 with rwo and strict-local volume
Given Create persistentvolumeclaim 0 using RWO volume
And Create persistentvolumeclaim 1 using RWX volume
And Create persistentvolumeclaim 2 using RWO volume with strict-local storageclass

And Create deployment 0 with persistentvolumeclaim 0
And Create deployment 1 with persistentvolumeclaim 1
And Create deployment 2 with persistentvolumeclaim 2

And Create statefulset 0 using RWO volume
And Create statefulset 1 using RWX volume
And Create statefulset 2 using RWO volume with strict-local storageclass

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to deployment 0
And Keep writing data to deployment 1
And Keep writing data to deployment 2
And Keep writing data to statefulset 0
And Keep writing data to statefulset 1
And Keep writing data to statefulset 2
And Keep writing data to pod of deployment 0
And Keep writing data to pod of deployment 1
And Keep writing data to pod of deployment 2
And Keep writing data to pod of statefulset 0
And Keep writing data to pod of statefulset 1
And Keep writing data to pod of statefulset 2

When Restart all worker nodes
And Wait for longhorn ready
And Wait for deployment 0 pods stable
And Wait for deployment 1 pods stable
And Wait for deployment 2 pods stable
And Wait for statefulset 0 pods stable
And Wait for statefulset 1 pods stable
And Wait for statefulset 2 pods stable

Then Check deployment 0 works
And Check deployment 1 works
Expand All @@ -105,23 +141,34 @@ Reboot All Worker Nodes While Workload Heavy Writing
END

Power Off All Worker Nodes For More Than Pod Eviction Timeout While Workload Heavy Writing
Given Create deployment 0 with rwo volume
And Create deployment 1 with rwx volume
And Create deployment 2 with rwo and strict-local volume
And Create statefulset 0 with rwo volume
And Create statefulset 1 with rwx volume
And Create statefulset 2 with rwo and strict-local volume
Given Create persistentvolumeclaim 0 using RWO volume
And Create persistentvolumeclaim 1 using RWX volume
And Create persistentvolumeclaim 2 using RWO volume with strict-local storageclass

And Create deployment 0 with persistentvolumeclaim 0
And Create deployment 1 with persistentvolumeclaim 1
And Create deployment 2 with persistentvolumeclaim 2

And Create statefulset 0 using RWO volume
And Create statefulset 1 using RWX volume
And Create statefulset 2 using RWO volume with strict-local storageclass

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to deployment 0
And Keep writing data to deployment 1
And Keep writing data to deployment 2
And Keep writing data to statefulset 0
And Keep writing data to statefulset 1
And Keep writing data to statefulset 2
And Keep writing data to pod of deployment 0
And Keep writing data to pod of deployment 1
And Keep writing data to pod of deployment 2
And Keep writing data to pod of statefulset 0
And Keep writing data to pod of statefulset 1
And Keep writing data to pod of statefulset 2

When Power off all worker nodes for 6 mins
And Wait for longhorn ready
And Wait for deployment 0 pods stable
And Wait for deployment 1 pods stable
And Wait for deployment 2 pods stable
And Wait for statefulset 0 pods stable
And Wait for statefulset 1 pods stable
And Wait for statefulset 2 pods stable

Then Check deployment 0 works
And Check deployment 1 works
Expand All @@ -132,33 +179,35 @@ Power Off All Worker Nodes For More Than Pod Eviction Timeout While Workload Hea
END

Reboot Volume Node While Workload Heavy Writing
Given Create statefulset 0 with ${VOLUME_TYPE} volume
Given Create statefulset 0 using ${VOLUME_TYPE} volume
FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to statefulset 0
And Keep writing data to pod of statefulset 0

When Reboot volume node of statefulset 0
And Wait for volume of statefulset 0 healthy
And Wait for statefulset 0 stable
And Wait for statefulset 0 pods stable

Then Check statefulset 0 works
END

Power Off Volume Node For More Than Pod Eviction Timeout While Workload Heavy Writing
Given Create statefulset 0 with rwo volume
Given Create statefulset 0 using RWO volume

FOR ${i} IN RANGE ${LOOP_COUNT}
And Keep writing data to statefulset 0
And Keep writing data to pod of statefulset 0

When Power off volume node of statefulset 0 for 6 mins
When Power off volume node of statefulset 0 for 6 minutes
And Wait for volume of statefulset 0 healthy
And Wait for statefulset 0 stable
And Wait for statefulset 0 pods stable

Then Check statefulset 0 works
END

Reboot Volume Node While Heavy Writing And Recurring Jobs Exist
Given Create volume 0 with 2 GB and 1 replicas
And Create volume 1 with 2 GB and 3 replicas
And Attach volume 0
And Attach volume 1
And Keep writing data to volume 0
And Keep Writing data to volume 1
And Create snapshot and backup recurringjob for volume 0
Expand All @@ -176,6 +225,8 @@ Reboot Volume Node While Heavy Writing And Recurring Jobs Exist
Reboot Replica Node While Heavy Writing And Recurring Jobs Exist
Given Create volume 0 with 2 GB and 1 replicas
And Create volume 1 with 2 GB and 3 replicas
And Attach volume 0
And Attach volume 1
And Keep Writing data to volume 0
And Keep Writing data to volume 1
And Create snapshot and backup recurringjob for volume 0
Expand Down
Loading

0 comments on commit 724833e

Please sign in to comment.