add RDMA mode configuration example in readme

Signed-off-by: Sebastian Sch <sebassch@gmail.com>

Author: SchSeba

README.md

@@ -301,32 +301,6 @@ spec:
 
 > **NOTE**: Currently only `mellanox` plugin can be disabled.
 
-### Parallel draining
-
-It is possible to drain more than one node at a time using this operator.
-
-The configuration is done via the SriovNetworkNodePool, selecting a number of nodes using the node selector and how many
-nodes in parallel from the pool the operator can drain in parallel. maxUnavailable can be a number or a percentage.
-
-> **NOTE**: every node can only be part of one pool, if a node is selected by more than one pool, then it will not be drained
-
-> **NOTE**: If a node is not part of any pool it will have a default configuration of maxUnavailable 1
-
-**Example**:
-
-```yaml
-apiVersion: sriovnetwork.openshift.io/v1
-kind: SriovNetworkPoolConfig
-metadata:
-  name: worker
-  namespace: sriov-network-operator
-spec:
-  maxUnavailable: 2
-  nodeSelector:
-    matchLabels:
-      node-role.kubernetes.io/worker: ""
-```
-
 ## Feature Gates
 
 Feature gates are used to enable or disable specific features in the operator.
@@ -371,6 +345,115 @@ spec:
   ...
 ```
 
+## SriovNetworkPoolConfig Configuration
+
+The `SriovNetworkPoolConfig` CRD provides advanced configuration capabilities for managing groups of nodes in SR-IOV network environments. This custom resource allows cluster administrators to define node-level configuration policies that apply to specific sets of nodes selected by label selectors.
+
+### Purpose and Benefits
+
+The `SriovNetworkPoolConfig` CRD serves multiple purposes:
+
+1. **Node Pool Management**: Groups nodes into logical pools for coordinated configuration updates
+2. **Parallel Operations**: Enables controlled parallel draining and configuration updates across multiple nodes
+3. **RDMA Configuration**: Provides centralized RDMA mode configuration for selected nodes
+
+### Key Configuration Fields
+
+#### Node Selection and Availability Control
+
+- **nodeSelector**: Specifies which nodes belong to this pool using Kubernetes label selectors
+- **maxUnavailable**: Controls how many nodes can be unavailable simultaneously during updates (supports both integer and percentage values)
+
+#### RDMA Mode Configuration
+
+The `rdmaMode` field allows you to configure the RDMA (Remote Direct Memory Access) subsystem behavior for all nodes in the pool:
+
+- **shared**: Multiple processes can share RDMA resources simultaneously
+- **exclusive**: RDMA resources are exclusively assigned to a single process
+
+### Parallel Draining
+
+It is possible to drain more than one node at a time using this operator.
+
+The configuration is done via the SriovNetworkPoolConfig, selecting a number of nodes using the node selector and how many
+nodes in parallel from the pool the operator can drain in parallel. maxUnavailable can be a number or a percentage.
+
+> **NOTE**: every node can only be part of one pool, if a node is selected by more than one pool, then it will not be drained
+
+> **NOTE**: If a node is not part of any pool it will have a default configuration of maxUnavailable 1
+
+### Configuration Examples
+
+#### Basic Parallel Draining Configuration
+
+```yaml
+apiVersion: sriovnetwork.openshift.io/v1
+kind: SriovNetworkPoolConfig
+metadata:
+  name: worker
+  namespace: sriov-network-operator
+spec:
+  maxUnavailable: 2
+  nodeSelector:
+    matchLabels:
+      node-role.kubernetes.io/worker: ""
+```
+
+### RDMA Mode
+
+The RDMA mode setting affects how RDMA resources are managed across the nodes in the pool.
+The RDMA mode configuration is applied during node configuration updates and affects all The Mellanox SR-IOV devices on the selected nodes.
+
+*NOTE:* swtiching rdma mode will trigger a reboot to all the nodes in the pool base on the maxUnavailable configuration
+
+#### Exclusive RDMA Mode Configuration
+
+```yaml
+apiVersion: sriovnetwork.openshift.io/v1
+kind: SriovNetworkPoolConfig
+metadata:
+  name: rdma-workers
+  namespace: sriov-network-operator
+spec:
+  maxUnavailable: 1
+  rdmaMode: exclusive
+  nodeSelector:
+    matchLabels:
+      node-role.kubernetes.io/worker: ""
+```
+
+#### SriovNetwork with RDMA CNI Plugin
+
+When RDMA mode is set to exclusive, you can create an SriovNetwork that injects the RDMA CNI plugin to allow pods to access hardware counters. Here's an example configuration:
+
+```yaml
+apiVersion: sriovnetwork.openshift.io/v1
+kind: SriovNetwork
+metadata:
+  name: rdma-network
+  namespace: sriov-network-operator
+spec:
+  resourceName: rdma_shared_device_a
+  networkNamespace: default
+  ipam: |
+    {
+      "type": "host-local",
+      "subnet": "10.10.10.0/24",
+      "rangeStart": "10.10.10.171",
+      "rangeEnd": "10.10.10.181"
+    }
+  metaPluginsConfig: |
+    {
+      "type": "rdma"
+    }
+```
+
+This configuration:
+- Uses `metaPluginsConfig` to inject the RDMA CNI plugin
+- Allows pods using this network to access hardware counters
+- Requires the nodes to be configured with RDMA mode set to exclusive
+- Works with SR-IOV network policies that have `isRdma: true` specified
+
 ## Components and design
 
 This operator is split into 2 components: