Skip to content

Commit e638ab5

Browse files
DangerOnTheRangercici37
authored andcommitted
Reword resource constraint section.
1 parent 46d35e2 commit e638ab5

File tree

1 file changed

+109
-14
lines changed

1 file changed

+109
-14
lines changed

content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md

Lines changed: 109 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1051,20 +1051,115 @@ A portion of the schema is correlatable if all `array` parent schemas are of typ
10511051

10521052
#### Resource Constraints
10531053

1054-
CEL expressions have the potential to consume unacceptable amounts of API server resources. We constrain the resource utilization in following ways:
1055-
- Validation of CEL expression's "cost" when a CEL expression is written to a field in a CRD (at CRD creation/update time)
1056-
- Runtime cost budget during CEL evaluation
1057-
- CEL validation might fail due to runtime cost budget exceed with error message `validation failed due to running out of cost budget, no further validation rules will be run`
1058-
- CEL validation might fail due to cost limit exceed per expression with message `operation cancelled: actual cost limit exceeded: no further validation rules will be run due to call cost exceeds limit for rule:{$rule}`
1059-
- Go context cancellation to bound CEL expression evaluation to the request lifetime
1060-
1061-
Guidelines for working with estimated limits:
1062-
- Adding MaxItems, MaxProperties and MaxLength limits on all data accessed by CEL rules is the best practice.
1063-
- O(n) - For simple rules, it is possible to iterate across a single map/list/string without exceeding the limit, but adding limits on all data accessed by CEL rules is the best practice
1064-
- O(n^2)+ the product of the max lengths usually needs to be <1,000,000. E.g. 1000 for 2 levels of nesting, 100 for 3 levels of nesting
1065-
- O(n^3) - should generally be avoided
1066-
1067-
// TODO: edit info for cost estimation
1054+
Resource consumption of validation rules is checked at CustomResourceDefinition creation and update time. If a rule is estimated to be prohibitively expensive to execute, it will result in a validation error. A similar
1055+
system is used at runtime that observes the actions the interpreter takes. If the interpreter executes
1056+
too many instructions, execution of the rule will be halted, and an error will result.
1057+
Each CustomResourceDefinition is also allowed a certain amount of resources to finish executing all of
1058+
its validation rules. If the sum total of its rules are estimated at creation time to go over that limit,
1059+
then a validation error will also occur.
1060+
1061+
In general, both systems will allow rules that do not need to iterate; these rules will
1062+
always take the same amount of time regardless of how large their input is. `self.foo == 1` will be allowed.
1063+
But if `foo` is a string and we instead have `self.foo.contains("someString")`, our rule will take
1064+
longer to execute depending on how long `foo` is. Another example would be if `foo` was an array, and we
1065+
had a rule `self.foo.all(x, x > 5)`. The cost system will always assume the worst-case scenario if
1066+
a limit on the length of `foo` is not given, and this will happen for anything that can be iterated
1067+
over (lists, maps, etc.).
1068+
1069+
Because of this, it is considered best practice to put a limit via `maxItems`, `maxProperties`, and
1070+
`maxLength` for anything that will be processed in a validation rule in order to prevent validation errors during cost estimation. For example, given this schema with one rule:
1071+
1072+
```yaml
1073+
openAPIV3Schema:
1074+
type: object
1075+
properties:
1076+
foo:
1077+
type: array
1078+
items:
1079+
type: string
1080+
x-kubernetes-validations:
1081+
- rule: "self.all(x, x.contains('a string'))"
1082+
```
1083+
1084+
The cost system will not allow this rule. Using `self.all` means calling `contains` on every string in `foo`,
1085+
which in turn will check the given string to see if it contains `'a string'`. Without limits, this is a very
1086+
expensive rule:
1087+
1088+
```
1089+
spec.validation.openAPIV3Schema.properties[spec].properties[foo].x-kubernetes-validations[0].rule: Forbidden:
1090+
CEL rule exceeded budget by more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and
1091+
maxLength where arrays, maps, and strings are used)
1092+
```
1093+
1094+
Without limits being set, the estimated cost of this rule will exceed the per-rule cost limit. But if we
1095+
add limits in the appropriate places, the rule will be allowed:
1096+
1097+
```yaml
1098+
openAPIV3Schema:
1099+
type: object
1100+
properties:
1101+
foo:
1102+
type: array
1103+
maxItems: 25
1104+
items:
1105+
type: string
1106+
maxLength: 10
1107+
x-kubernetes-validations:
1108+
- rule: "self.all(x, x.contains('a string'))"
1109+
```
1110+
1111+
The cost estimation system takes into account how many times the rule will be executed in addition to the
1112+
estimated cost of the rule itself. For instance, the following rule will have the same estimated cost as the
1113+
previous example (despite the rule now being defined on the individual array items):
1114+
1115+
```yaml
1116+
openAPIV3Schema:
1117+
type: object
1118+
properties:
1119+
foo:
1120+
type: array
1121+
maxItems: 25
1122+
items:
1123+
type: string
1124+
x-kubernetes-validations:
1125+
- rule: "self.contains('a string'))"
1126+
maxLength: 10
1127+
```
1128+
1129+
If a list inside of a list has a validation rule that uses `self.all`, that is significantly more expensive
1130+
than a non-nested list with the same rule. A rule that would have been allowed on a non-nested list might need lower limits set on both nested lists in order to be allowed. For example, even without having limits set,
1131+
the following rule is allowed:
1132+
1133+
```yaml
1134+
openAPIV3Schema:
1135+
type: object
1136+
properties:
1137+
foo:
1138+
type: array
1139+
items:
1140+
type: integer
1141+
x-kubernetes-validations:
1142+
- rule: "self.all(x, x == 5)"
1143+
```
1144+
1145+
But the same rule on the following schema (with a nested array added) produces a validation error:
1146+
1147+
```yaml
1148+
openAPIV3Schema:
1149+
type: object
1150+
properties:
1151+
foo:
1152+
type: array
1153+
items:
1154+
type: array
1155+
items:
1156+
type: integer
1157+
x-kubernetes-validations:
1158+
- rule: "self.all(x, x == 5)"
1159+
```
1160+
1161+
This is because each item of `foo` is itself an array, and each subarray in turn calls `self.all`. Avoid nested
1162+
lists and maps if possible where validation rules are used.
10681163

10691164
### Defaulting
10701165

0 commit comments

Comments
 (0)