add test example for network loss, reorder, duplicate and corrupt in TiDB

README.md

@@ -32,10 +32,10 @@ See [fotmat of record](./format.md) for details.
 - [Network Delay](./network/delay.md)
 - [Network Bandwidth](./network/bandwidth.md)
 - [Network Partition](./network/partition.md)
-- Network Packet Loss
-- Network Packet Reorder
-- Network Packet Duplicate
-- Network Packet Corrupt
+- [Network Packet Loss](./network/loss.md)
+- [Network Packet Reorder](./network/reorder.md)
+- [Network Packet Duplicate](./network/duplicate.md)
+- [Network Packet Corrupt](./network/corrupt.md)
 
 ### IO
 

network/bandwidth.md

@@ -4,7 +4,7 @@
 
 ### Description
 
-Test the availability of TiDB cluster in network bandwidth scenarios.
+Test the availability of the TiDB cluster in network bandwidth scenarios.
 
 ### Hypothesis
 
@@ -75,7 +75,7 @@ You can test more scenarios by using Chaos Mesh. For example:
 - Limiting the network bandwidth between TiDB nodes.
 - Limiting the network bandwidth between TiKV nodes.
 
-All you need to do is adjust the `selector` and `target` in the YAML configuration. For limiting the network bandwidth between TiKV nodes, the YAML configuration looks like below:
+All you need to do is adjust the `selector` and `target` in the YAML configuration. For limiting the network bandwidth between TiKV nodes, the YAML configuration looks like the below:
 
 ```YAML
 kind: NetworkChaos

network/corrupt.md

@@ -0,0 +1,102 @@
+# Test Network Chaos on TiDB
+
+## Network Corrupt
+
+### Description
+
+Test the availability of the TiDB cluster in network package corrupt scenarios.
+
+### Hypothesis
+
+When the network package corruption occurs between TiDB and TiKV nodes, the QPS/TPS of TiDB will drop significantly, but it can still provide services normally.
+
+### Preparation
+
+1. Install chaos-mesh.
+2. A TiDB cluster on k8s with at least 3 TiKV Pods, deploy with [TiDB operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview).
+3. Running payload on TiDB cluster.
+
+### Quick start
+
+1. Chaos Mesh fault YAML configuration:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: tidb-cluster
+  name: network-corrupt
+spec:
+  selector:
+    namespaces:
+      - tidb-cluster
+    labelSelectors:
+      app.kubernetes.io/component: tidb
+  mode: all
+  action: corrupt
+  corrupt:
+    corrupt: '50'
+    correlation: '50'
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - tidb-cluster
+      labelSelectors:
+        app.kubernetes.io/component: tikv
+    mode: all
+```
+
+Saving the YAML configuration above into file network-corrupt.yaml.
+
+2. Using Kubectl to create the experiment:
+
+```
+kubectl create -f network-corrupt.yaml
+```
+
+3. Verifying TiDB's status:
+
+    Check QPS & TPS of TiDB in Grafana.
+    <!-- TODO: Add some Grafana picture -->
+
+4. Result:
+
+Judge whether the hypothesis is correct or not based on the results of the test process.
+
+### More example
+
+You can test more scenarios by using Chaos Mesh. For example:
+
+- Network package corruption occurs between TiDB and TiKV.
+- Network package corruption occurs between TiKV and PD.
+- Network package corruption occurs between TiKV nodes.
+- Network package corruption occurs between PD nodes.
+
+All you need to do is adjust the `selector` and `target` in the YAML configuration. To inject network package corrupt between TiKV nodes, the YAML configuration looks like the below:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: tidb-cluster
+  name: network-corrupt
+spec:
+  selector:
+    pods:
+      tidb-cluster:
+        - basic-tikv-0
+  mode: all
+  action: corrupt
+  corrupt:
+    corrupt: '50'
+    correlation: '50'
+  direction: both
+  target:
+    selector:
+      pods:
+        tidb-cluster:
+          - basic-tikv-1
+          - basic-tikv-2
+    mode: all
+```

network/delay.md

@@ -4,7 +4,7 @@
 
 ### Description
 
-Test the availability of TiDB cluster in network delay scenarios.
+Test the availability of the TiDB cluster in network delay scenarios.
 
 ### Hypothesis
 
@@ -75,7 +75,7 @@ You can test more scenarios by using Chaos Mesh. For example:
 - Increasing network delay between TiKV nodes.
 - Increasing network delay between PD nodes.
 
-All you need to do is adjust the `selector` and `target` in the YAML configuration. For increasing network delay between TiKV nodes, the YAML configuration looks like below:
+All you need to do is adjust the `selector` and `target` in the YAML configuration. For increasing network delay between TiKV nodes, the YAML configuration looks like the below:
 
 ```YAML
 kind: NetworkChaos

network/duplicate.md

@@ -0,0 +1,102 @@
+# Test Network Chaos on TiDB
+
+## Network Duplicate
+
+### Description
+
+Test the availability of the TiDB cluster in network package duplicate scenarios.
+
+### Hypothesis
+
+When the network package duplicate occurs between TiDB and TiKV nodes, the QPS/TPS of TiDB will drop significantly, but it can still provide services normally.
+
+### Preparation
+
+1. Install chaos-mesh.
+2. A TiDB cluster on k8s with at least 3 TiKV Pods, deploy with [TiDB operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview).
+3. Running payload on TiDB cluster.
+
+### Quick start
+
+1. Chaos Mesh fault YAML configuration:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: tidb-cluster
+  name: network-duplicate
+spec:
+  selector:
+    namespaces:
+      - tidb-cluster
+    labelSelectors:
+      app.kubernetes.io/component: tidb
+  mode: all
+  action: duplicate
+  duplicate:
+    duplicate: '50'
+    correlation: '50'
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - tidb-cluster
+      labelSelectors:
+        app.kubernetes.io/component: tikv
+    mode: all
+```
+
+Saving the YAML configuration above into file network-duplicate.yaml.
+
+2. Using Kubectl to create the experiment:
+
+```
+kubectl create -f network-duplicate.yaml
+```
+
+3. Verifying TiDB's status:
+
+    Check QPS & TPS of TiDB in Grafana.
+    <!-- TODO: Add some Grafana picture -->
+
+4. Result:
+
+Judge whether the hypothesis is correct or not based on the results of the test process.
+
+### More example
+
+You can test more scenarios by using Chaos Mesh. For example:
+
+- Network package duplicate occurs between TiDB and TiKV.
+- Network package duplicate occurs between TiKV and PD.
+- Network package duplicate occurs between TiKV nodes.
+- Network package duplicate occurs between PD nodes.
+
+All you need to do is adjust the `selector` and `target` in the YAML configuration. To inject network package duplicate between TiKV nodes, the YAML configuration looks like the below:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: tidb-cluster
+  name: network-duplicate
+spec:
+  selector:
+    pods:
+      tidb-cluster:
+        - basic-tikv-0
+  mode: all
+  action: duplicate
+  duplicate:
+    duplicate: '50'
+    correlation: '50'
+  direction: both
+  target:
+    selector:
+      pods:
+        tidb-cluster:
+          - basic-tikv-1
+          - basic-tikv-2
+    mode: all
+```

network/loss.md

@@ -0,0 +1,102 @@
+# Test Network Chaos on TiDB
+
+## Network Loss
+
+### Description
+
+Test the availability of TiDB cluster in network package loss scenarios.
+
+### Hypothesis
+
+When the network package loss occurs between TiKV nodes, the QPS/TPS of TiDB will drop significantly, but it can still provide services normally.
+
+### Preparation
+
+1. Install chaos-mesh.
+2. A TiDB cluster on k8s with at least 3 TiKV Pods, deploy with [TiDB operator](https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview).
+3. Running payload on TiDB cluster.
+
+### Quick start
+
+1. Chaos Mesh fault YAML configuration:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: chaos-testing
+  name: network-loss
+spec:
+  selector:
+    pods:
+      tidb-cluster:
+        - basic-tikv-0
+  mode: all
+  action: loss
+  loss:
+    loss: '10'
+    correlation: '50'
+  direction: to
+  target:
+    selector:
+      pods:
+        tidb-cluster:
+          - basic-tikv-1
+          - basic-tikv-2
+    mode: all
+```
+
+Saving the YAML configuration above into file network-loss.yaml.
+
+2. Using Kubectl to create the experiment:
+
+```
+kubectl create -f network-loss.yaml
+```
+
+3. Verifying TiDB's status:
+
+    Check QPS & TPS of TiDB in Grafana.
+    <!-- TODO: Add some Grafana picture -->
+
+4. Result:
+
+Judge whether the hypothesis is correct or not based on the results of the test process.
+
+### More example
+
+You can test more scenarios by using Chaos Mesh. For example:
+
+- Network package loss occurs between TiDB and TiKV.
+- Network package loss occurs between TiKV and PD.
+- Network package loss occurs between TiDB nodes.
+- Network package loss occurs between PD nodes.
+
+All you need to do is adjust the `selector` and `target` in the YAML configuration. For injecting network package loss between TiDB and TiKV, the YAML configuration looks like below:
+
+```YAML
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  namespace: chaos-testing
+  name: network-loss
+spec:
+  selector:
+    namespaces:
+      - tidb-cluster
+    labelSelectors:
+      app.kubernetes.io/component: tidb
+  mode: all
+  action: loss
+  loss:
+    loss: '10'
+    correlation: '50'
+  direction: to
+  target:
+    selector:
+      namespaces:
+        - tidb-cluster
+      labelSelectors:
+        app.kubernetes.io/component: tikv
+    mode: all
+```

network/partition.md

@@ -4,11 +4,11 @@
 
 ### Description
 
-Test the availability of TiDB cluster in network partition scenarios.
+Test the availability of the TiDB cluster in network partition scenarios.
 
 ### Hypothesis
 
-When a network partition occurs between a TiKV node and other TiKV nodes in the cluster, the QPS/TPS drop significantly, and then recover to normal levels, and the data is consistent.
+When a network partition occurs between a TiKV node and other TiKV nodes in the cluster, the QPS/TPS drop significantly and then recover to normal levels, and the data is consistent.
 
 ### Preparation
 
@@ -70,7 +70,7 @@ You can test more scenarios by using Chaos Mesh. For example:
 - Network partition occurs between TiDB and TiKV.
 - Network partition occurs between PD and TiKV.
 
-All you need to do is adjust the `selector` and `target` in the YAML configuration. For Network partition occurs between TiDB and TiKV, the YAML configuration looks like below:
+All you need to do is adjust the `selector` and `target` in the YAML configuration. For network partition that occurs between TiDB and TiKV, the YAML configuration looks like the below:
 
 ```YAML
 kind: NetworkChaos