Skip to content

Commit e73f047

Browse files
author
Vlade Lekic
committed
Initial commit.
1 parent dba9638 commit e73f047

File tree

2 files changed

+166
-0
lines changed

2 files changed

+166
-0
lines changed

‎articles/service-fabric/service-fabric-cluster-resource-manager-inbuild-throttling.md

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
---
2+
title: InBuild Throttling
3+
description: Configure, understand and apply InBuild Throttling constraint.
4+
ms.topic: conceptual
5+
ms.author: vladelekic
6+
author: vladelekic
7+
ms.service: service-fabric
8+
services: service-fabric
9+
ms.date: 02/28/2024
10+
---
11+
12+
# Throttling InBuild Replicas per Node
13+
14+
Replicas of stateful services transition through several phases during their lifecycle: InBuild, Ready, Closing, and Dropped states. Replicas in the Ready and Closing states can declare load, either default or dynamic load. Replicas in the Dropped state are irrelevant in terms of resource utilization. The resource utilization of InBuild (IB) replicas is specific, so the Cluster Resource Manager provides additional support for handling that utilization. Since IB replicas cannot report dynamic load, and their resource utilization might be higher than less utilized replicas in the Ready state, especially for I/O and memory-related metrics, the CRM allows limiting the number of concurrent builds per node.
15+
16+
The CRM provides support for limiting the number of IB replicas per node and defining the limit per node type or for all nodes that satisfy placement constraints. Depending on the constraint priority, according to general rules about constraint prioritization, the CRM will halt movements and creations of replicas on a node if the actions violate the defined IB limit for that node. The constraint blocks only operations that could cause high I/O and memory consumption during the InBuild phase, especially when extensively replicating context from other active replicas.
17+
18+
Movements of any kind and promotion of StandBy replicas are restricted operations that cause state replication and extensive resource utilization during the InBuild phase. On the other hand, the promotion of an active Secondary replica to Primary replica is not a problematic operation, so the constraint will not block such operations. During the promotion of a Secondary replica, the state of the replica is up-to-date with the Primary replica, eliminating the need for additional replication.
19+
20+
> [!NOTE]
21+
> The promotion of StandBy replicas could be blocked due to InBuild replicas per node throttling. The transition from StandBy to Ready replicas could cause extensive I/O and memory utilization, depending on the amount of context that needs to be replicated from active replicas. Thus, ignoring promotions of StandBy replicas could cause issues that the InBuild replicas per node throttling constraint aims to resolve.
22+
>
23+
24+
## Configuring the InBuild Replicas per Node Throttling
25+
26+
### Enable InBuild Replicas per Node Throttling for Action
27+
28+
There are three different categories of actions that the Cluster Resource Manager performs:
29+
30+
* _Placement_: Controls placement of missing replicas, orchestrates swaps during upgrades, and removal of additional replicas.
31+
* _Constraint Check_: Enforces rules.
32+
* _Balancing_: Performs actions that reduce the imbalance of node total utilization in a cluster.
33+
34+
The Cluster Resource Manager allows enabling/disabling throttling of IB replicas per node for each category. Configurations that control whether throttling is active for specific actions are *ThrottlePlacementPhase*, *ThrottleConstraintCheckPhase*, and *ThrottleBalancingPhase*, respectively. The value specified for these configurations is boolean. The cluster manifest section that explicitly defines these configurations is provided below:
35+
36+
```xml
37+
<Section Name="PlacementAndLoadBalancing">
38+
<Parameter Name="ThrottlePlacementPhase" Value="true">
39+
<Parameter Name="ThrottleConstraintCheckPhase" Value="true">
40+
<Parameter Name="ThrottleBalancingPhase" Value="true">
41+
</Section>
42+
```
43+
44+
Here's an example of configurations that enable IB replicas throttling per node, defined via ClusterConfig.json for standalone deployments or Template.json for Azure-hosted clusters:
45+
46+
```json
47+
"fabricSettings": [
48+
{
49+
"name": "PlacementAndLoadBalancing",
50+
"parameters": [
51+
{
52+
"name": "ThrottlePlacementPhase",
53+
"value": "true"
54+
},
55+
{
56+
"name": "ThrottleConstraintCheckPhase",
57+
"value": "true"
58+
},
59+
{
60+
"name": "ThrottleBalancingPhase",
61+
"value": "true"
62+
}
63+
]
64+
}
65+
]
66+
```
67+
68+
### Configure InBuild Replicas per Node Limits
69+
70+
The Cluster Resource Manager allows defining IB limits globally and for each category of actions:
71+
72+
* MaximumInBuildReplicasPerNode: Defines IB limits globally. These limits will be used to evaluate the final IB limit for each category.
73+
* MaximumInBuildReplicasPerNodePlacementThrottle: Defines IB limits for the placement category. These limits will be used to evaluate the final IB limit only for the placement category.
74+
* MaximumInBuildReplicasPerNodeConstraintCheckThrottle: Defines IB limits for the constraint check category. These limits will be used to evaluate the final IB limit only for the constraint check category.
75+
* MaximumInBuildReplicasPerNodeBalancingThrottle: Defines IB limits for the balancing category. These limits will be used to evaluate the final IB limit only for the balancing category.
76+
77+
For each option above, the Cluster Resource Manager provides an additional two options for defining the limit of IB replicas:
78+
79+
* Define IB limit for all nodes in a single node type.
80+
* Define IB limit for all nodes with a matching placement constraint.
81+
82+
These rules allow you to define multiple values for a single category, and the CRM will always respect the most strict limit that you provided. The limit for each node in a specific phase is the lowest value according to node type or any placement property that corresponds to that node, for both global limits and category limits. If the limit for an action category for a specific node is not defined, the CRM will assume that there is no upper IB replica count for a node.
83+
84+
The cluster manifest sections that explicitly define limits for each phase are provided below:
85+
86+
```xml
87+
<Section Name="MaximumInBuildReplicasPerNode">
88+
<Parameter Name="NodeTypeA" Value="10" />
89+
<Parameter Name="NodeTypeB" Value="20" />
90+
<Parameter Name="NodeTypeName == NodeTypeA || NodeTypeName == NodeTypeC" Value="15" />
91+
</Section>
92+
93+
<Section Name="MaximumInBuildReplicasPerNodePlacementThrottle">
94+
<Parameter Name="NodeTypeC" Value="20" />
95+
</Section>
96+
97+
<Section Name="MaximumInBuildReplicasPerNodeConstraintCheckThrottle">
98+
<Parameter Name="NodeTypeD" Value="10" />
99+
<Parameter Name="Color == Blue" Value="8" />
100+
</Section>
101+
102+
<Section Name="MaximumInBuildReplicasPerNodeBalancingThrottle">
103+
<Parameter Name="Color == Red" Value="25" />
104+
</Section>
105+
```
106+
107+
Here's an example of the same IB limits defined via ClusterConfig.json for standalone deployments or Template.json for Azure-hosted clusters:
108+
109+
```json
110+
"fabricSettings": [
111+
{
112+
"name": "MaximumInBuildReplicasPerNode",
113+
"parameters": [
114+
{
115+
"name": "NodeTypeA",
116+
"value": "10"
117+
},
118+
{
119+
"name": "NodeTypeB",
120+
"value": "20"
121+
},
122+
{
123+
"name": "NodeTypeName == NodeTypeA || NodeTypeName == NodeTypeC",
124+
"value": "15"
125+
}
126+
]
127+
},
128+
{
129+
"name": "MaximumInBuildReplicasPerNodePlacementThrottle",
130+
"parameters": [
131+
{
132+
"name": "NodeTypeC",
133+
"value": "20"
134+
}
135+
]
136+
},
137+
{
138+
"name": "MaximumInBuildReplicasPerNodeConstraintCheckThrottle",
139+
"parameters": [
140+
{
141+
"name": "NodeTypeD",
142+
"value": "10"
143+
},
144+
{
145+
"name": "Color == Blue",
146+
"value": "8"
147+
}
148+
]
149+
},
150+
{
151+
"name": "MaximumInBuildReplicasPerNodeBalancingThrottle",
152+
"parameters": [
153+
{
154+
"name": "Color == Red",
155+
"value": "25"
156+
}
157+
]
158+
}
159+
]
160+
```
161+
162+
## Next Steps
163+
- For more information about replica states, check out the article on [replica lifecycle](service-fabric-concepts-replica-lifecycle.md)
164+
- For more information about balancing and other action categories, check out the article on [balancing action](service-fabric-cluster-resource-manager-balancing.md)

‎articles/service-fabric/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,8 @@
379379
href: service-fabric-cluster-resource-manager-autoscaling.md
380380
- name: Node tagging
381381
href: service-fabric-cluster-resource-manager-node-tagging.md
382+
- name: InBuild replicas per node throttling
383+
href: service-fabric-cluster-resource-manager-inbuild-throttling.md
382384
- name: Monitoring and diagnostics
383385
items:
384386
- name: Monitoring overview

0 commit comments

Comments
 (0)