Skip to content

Commit

Permalink
Edits to BigAnimal PR5422
Browse files Browse the repository at this point in the history
  • Loading branch information
ebgitelman committed Jun 3, 2024
1 parent 51a2319 commit b452dc7
Showing 1 changed file with 15 additions and 14 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ navigation:
- Fault injection testing
---

You can test the fault tolerance of your cluster by deleting a VM in order to inject a fault. Once a VM is deleted, you can monitor
You can test the fault tolerance of your cluster by deleting a VM to inject a fault. Once a VM is deleted, you can monitor
the availability and recovery of the cluster.

## Requirements

Ensure you meet the following requirements before using fault injection testing:
Before using fault injection testing, ensure you meet the following requirements:

+ You have connected your BigAnimal cloud account with your Azure subscription. See [Setting up your Azure Marketplace account](/biganimal/latest/getting_started/02_azure_market_setup/) for more information.
+ You've connected your BigAnimal cloud account with your Azure subscription. See [Setting up your Azure Marketplace account](/biganimal/latest/getting_started/02_azure_market_setup/) for more information.
+ You have permissions in your Azure subscription to view and delete VMs and also the ability to view Kubernetes pods via Azure Kubernetes Service RBAC Reader.
+ You have PGD CLI installed. See [Installing PGD CLI](/pgd/latest/cli/installing_cli/#) for more information.
+ You have created a `pgd-cli-config.yml` file in your home directory. See [Configuring PGD CLI](/pgd/latest/cli/configuring_cli/) for more information.
+ You've created a `pgd-cli-config.yml` file in your home directory. See [Configuring PGD CLI](/pgd/latest/cli/configuring_cli/) for more information.

## Fault injection testing steps

Expand All @@ -26,9 +26,9 @@ Fault injection testing consists of the following steps:
3. Deleting a write leader node from your cluster
4. Monitoring cluster health

### Verifying Cluster Health
### Verifying cluster health

Use the following commands to monitor your cluster health, node info, raft, replication lag, and write leads.
Use the following commands to monitor your cluster health, node info, raft, replication lag, and write leads:

```shell
pgd check-health -f pgd-cli-config.yml
Expand Down Expand Up @@ -57,6 +57,7 @@ pgd help show-nodes

### Determining the write leader node for your cluster

This example shows the command for determining the write leader node for a cluster:

```shell
pgd show-groups -f pgd-cli-config.yml
Expand All @@ -68,27 +69,28 @@ p-x67kjp3fsq-a 2456382099 data world p-x67kjp3fsq-a-1
p-x67kjp3fsq-c 4147262499 data world
p-x67kjp3fsq-d 3176957154 data world p-x67kjp3fsq-d-1
```
In this example, the write leader node is **p-x67kjp3fsq-a-1**.
In this example, the write leader node is `p-x67kjp3fsq-a-1`.


## Deleting a write leader node from your cluster

To delete a write lead node from the cluster:
1. Log into BigAnimal.
2. In a separate browser window, log into your Microsoft Azure subscription.
3. In the left navigation of BigAnimal portal, choose **Clusters**.
4. Choose the cluster to test fault injection with and copy the string value from the URL. The string value is located after the underscore.
3. In the left navigation of BigAnimal portal, select **Clusters**.
4. Select the cluster to test fault injection with and copy the string value from the URL. The string value is located after the underscore.

![Delete a write lead](images/biganimal_faultinjectiontest_1.png)


5. In your Azure subscription, paste the string into the search and prefix it with **dp-** to search for the data plane.
* From the results, choose the Kubernetes service from the Azure Region that your cluster is deployed in.
5. To search for the data plane, in your Azure subscription, paste the string into the search and prefix it with `dp-`.

6. From the results, select the Kubernetes service from the Azure region that your cluster is deployed in.

![Delete a write lead 2](images/biganimal_faultinjectiontest_2.png)


6. Identify the Kubernetes service for your cluster.
7. Identify the Kubernetes service for your cluster.

![Delete a write lead](images/biganimal_faultinjectiontest_4.png)

Expand All @@ -97,11 +99,10 @@ To delete a write lead node from the cluster:
Don't delete the Azure Kubernetes VMSS here or sub resources directly.
!!!

7. Browse to the Data Plane, choose Workloads, and locate the Kubernetes resources for your cluster to delete a chosen node.
7. To delete a chosen node, browse to the data plane, select **Workloads**, and locate the Kubernetes resources for your cluster.
![Delete a write lead 3](images/biganimal_faultinjectiontest_3.png)


### Monitoring cluster health

After deleting a cluster node, you can monitor the health of the cluster using the same PGD CLI commands that you used to verify cluster health.

0 comments on commit b452dc7

Please sign in to comment.