Skip to content

Commit 6633845

Browse files
buehlerbuehler
authored
feat(webhooks): Add validation webhooks and cert support. (#136)
This closes #3. This adds support for validation webhooks. Validation webhooks may be implemented anywhere with the according `IValidationWebhook{TEntity}` interface. When registered to the operator builder, they perform validation on entities. During build time, a CA certificate is created and during container startup, the corresponding server certificate for the instance. Signed-off-by: Christoph Bühler <[email protected]>
1 parent 3537772 commit 6633845

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+1859
-440
lines changed

‎config/CodeAnalysis.targets‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
<Project>
22

33
<ItemGroup>
4-
<PackageReference Include="Roslynator.Analyzers" Version="3.0.0" PrivateAssets="All"/>
5-
<PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.261" PrivateAssets="All"/>
6-
<AdditionalFiles Include="$(MSBuildThisFileDirectory)/stylecop.json"/>
4+
<PackageReference Include="Roslynator.Analyzers" Version="3.0.0" PrivateAssets="All" />
5+
<PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.261" PrivateAssets="All" />
6+
<AdditionalFiles Include="$(MSBuildThisFileDirectory)/stylecop.json" />
77
</ItemGroup>
88

99
<PropertyGroup>

‎config/Common.targets‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,4 @@
1818
</AssemblyAttribute>
1919
</ItemGroup>
2020

21-
</Project>
21+
</Project>

‎docs/docfx.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
"dest": "public",
4040
"sitemap": {
4141
"baseUrl": "https://buehler.github.io/dotnet-operator-sdk/"
42-
}
42+
},
43+
"template":["default","templates/discordfx"]
4344
}
4445
}

‎docs/docs/events.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Events
1+
# Events / Event Series
22

33
Kubernetes knows "Events" which can be sort of attached to a resource
44
(i.e. a Kubernetes object).

‎docs/docs/toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@
1212
href: events.md
1313
- name: Finalizer
1414
href: finalizer.md
15+
- name: Webhooks
16+
href: webhooks.md
1517
- name: Utilities
1618
href: utilities.md
1719
- name: CLI Commands

‎docs/docs/webhooks.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Webhooks
2+
3+
Kubernetes supports various webhooks to extend the normal api behaviour
4+
of the master api. Those are documented on the
5+
[kubernetes website](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/).
6+
7+
`KubeOps` supports the following webhooks out of the box:
8+
- Validator / Validation
9+
10+
The following documentation should give the user an overview
11+
on how to implement a webhook what this implies to the written operator.
12+
13+
## General
14+
15+
In general, if your operator contains _any_ registered (registered in the
16+
DI) the build process that is provided via `KubeOps.targets` will
17+
generate a CA certificate for you.
18+
19+
So if you add a webhook to your operator the following changes
20+
to the normal deployment of the operator will happen:
21+
1. During "after build" phase, the sdk will generate
22+
a CA-certificate for self signed certificates for you.
23+
2. The ca certificate and the corresponding key are added
24+
to the deployment via kustomization config.
25+
3. A special config is added to the deployment via
26+
kustomization to use https.
27+
4. The deployment of the operator now contains an `init-container`
28+
that loads the `ca.pem` and `ca-key.pem` files and creates
29+
a server certificate. Also, a service and the corresponding
30+
webhook configurations are created.
31+
5. When the operator starts, an additional https route is registered
32+
with the created server certificate.
33+
34+
When a webhook is registered, the specified operations will
35+
trigger a POST call to the operator.
36+
37+
> [!NOTE]
38+
> Make sure you commit the `ca.pem` / `ca-key.pem` file.
39+
> During operator startup (init container) those files
40+
> are needed. Since this represents a self signed certificate,
41+
> and it is only used for cluster internal communication,
42+
> it is no security issue to the system. The service is not
43+
> exposed to the internet.
44+
45+
> [!NOTE]
46+
> The `server.pem` and `server-key.pem` files are generated
47+
> in the init container during pod startup.
48+
> Each pod / instance of the operator gets its own server
49+
> certificate but the CA must be shared among them.
50+
51+
## Local development
52+
53+
It is possible to test webhooks locally. For this, you need
54+
to register the webhook via dependency injection with the corresponding
55+
method (in the builder) and then start your operator.
56+
57+
The operator will run on a specific http address, depending on your
58+
configuration.
59+
Now, use [ngrok](https://ngrok.com/) or
60+
[localtunnel](https://localtunnel.github.io/www/) or something
61+
similar to create a HTTPS tunnel to your local running operator.
62+
63+
Now you can use the cli command of the sdk
64+
`dotnet run -- webhooks register --base-url <<TUNNEL URL>>` to
65+
register the webhooks under the tunnel's url.
66+
67+
The result is your webhook being called by the kubernetes api.
68+
It is suggested one uses `Docker Desktop` with kubernetes.
69+
70+
## Validation webhook
71+
72+
The general idea of this webhook type is to validate an entity
73+
before it is definitely created / updated or deleted.
74+
75+
The implementation of a webhook is fairly simple:
76+
- Create a class somewhere in your project.
77+
- Implement the `IValidationWebhook{TEntity}` interface.
78+
- Define the `Operations` (from the interface) that the validator
79+
is interested in.
80+
- Overwrite the corresponding methods.
81+
- Register it in the `IOperatorBuilder` with `AddValidationWebhook`.
82+
83+
> [!WARNING]
84+
> The interface contains default implementations for _ALL_ methods.
85+
> The default of the async methods are to call the sync ones.
86+
> The default of the sync methods is to return a "not implemented"
87+
> result.
88+
> The async methods take precedence over the synchronous ones.
89+
90+
The return value of the validation methods are `ValidationResult`
91+
objects. A result contains a boolean flag if the entity / operation
92+
is valid or not. It may contain additional warnings (if it is valid)
93+
that are presented to the user if the kubernetes api supports it.
94+
If the result is invalid, one may add a custom http status code
95+
as well as a custom error message that is presented to the user.
96+
97+
### Example
98+
99+
```c#
100+
public class TestValidator : IValidationWebhook<V2TestEntity>
101+
{
102+
public ValidatedOperations Operations => ValidatedOperations.Create | ValidatedOperations.Update;
103+
104+
public ValidationResult Create(V2TestEntity newEntity, bool dryRun) =>
105+
CheckSpec(newEntity)
106+
? ValidationResult.Success("The username may not be foobar.")
107+
: ValidationResult.Fail(StatusCodes.Status400BadRequest, @"Username is ""foobar"".");
108+
109+
public ValidationResult Update(V2TestEntity _, V2TestEntity newEntity, bool dryRun) =>
110+
CheckSpec(newEntity)
111+
? ValidationResult.Success("The username may not be foobar.")
112+
: ValidationResult.Fail(StatusCodes.Status400BadRequest, @"Username is ""foobar"".");
113+
114+
private static bool CheckSpec(V2TestEntity entity) => entity.Spec.Username != "foobar";
115+
}
116+
```
117+
118+
And then register the webhook in `Startup.cs`:
119+
120+
```c#
121+
public void ConfigureServices(IServiceCollection services)
122+
{
123+
services
124+
.AddKubernetesOperator()
125+
// ...
126+
.AddValidationWebhook<TestValidator>();
127+
}
128+
```

‎src/KubeOps/Build/KubeOps.targets‎

Lines changed: 61 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,96 +1,98 @@
1-
<Project DefaultTargets="GenerateAfterBuild">
2-
<Target Name="GenerateDockerfile">
1+
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="GenerateAfterBuild">
2+
<Target Name="BaseConfig">
3+
<PropertyGroup>
4+
<!-- Configuration for the base -->
5+
<KubeOpsBasePath Condition="'$(KubeOpsBasePath)' == ''">$(MSBuildProjectDirectory)</KubeOpsBasePath>
6+
</PropertyGroup>
7+
8+
<PropertyGroup>
9+
<!-- Configuration for the pathes where to store the generated yamls and elements -->
10+
<KubeOpsConfigRoot Condition="'$(KubeOpsConfigRoot)' == ''">$(KubeOpsBasePath)\config</KubeOpsConfigRoot>
11+
</PropertyGroup>
12+
313
<PropertyGroup>
414
<!-- Configuration for Docker related commands -->
515
<KubeOpsDockerfilePath Condition="'$(KubeOpsDockerfilePath)' == ''">$(KubeOpsBasePath)\Dockerfile</KubeOpsDockerfilePath>
616
<KubeOpsDockerTag Condition="'$(KubeOpsDockerTag)' == ''">latest</KubeOpsDockerTag>
717
</PropertyGroup>
818

9-
<Message Text="Generating Dockerfile" Importance="high"/>
10-
<Message Text="Dockerfile path: $(KubeOpsDockerfilePath)" Importance="normal"/>
11-
12-
<Message Condition="Exists('$(KubeOpsDockerfilePath)')" Text="Dockerfile already exists. Don't overwrite."
13-
Importance="high"/>
14-
<Exec Condition="!Exists('$(KubeOpsDockerfilePath)')"
15-
Command="dotnet $(OutputPath)$(TargetFileName) generator docker --out $(KubeOpsDockerfilePath) --dotnet-tag $(KubeOpsDockerTag) --solution-dir $(SolutionDir) --target-file $(TargetFileName) --project-path $(ProjectPath)"/>
16-
</Target>
17-
18-
<Target Name="GenerateCrds">
1919
<PropertyGroup>
2020
<!-- Configuration for the crd generation -->
2121
<KubeOpsCrdDir Condition="'$(KubeOpsCrdDir)' == ''">$(KubeOpsConfigRoot)\crds</KubeOpsCrdDir>
2222
<KubeOpsCrdFormat Condition="'$(KubeOpsCrdFormat)' == ''">Yaml</KubeOpsCrdFormat>
2323
<KubeOpsCrdUseOldCrds Condition="'$(KubeOpsCrdUseOldCrds)' == ''">false</KubeOpsCrdUseOldCrds>
2424
</PropertyGroup>
2525

26-
<Message Text="Generating CRDs" Importance="high"/>
27-
<Message Text="Configuration path: $(KubeOpsCrdDir)" Importance="normal"/>
28-
29-
<Exec Condition="'$(KubeOpsCrdUseOldCrds)' == 'false'"
30-
Command="dotnet $(OutputPath)$(TargetFileName) generator crds --out $(KubeOpsCrdDir) --format $(KubeOpsCrdFormat)"/>
31-
<Exec Condition="'$(KubeOpsCrdUseOldCrds)' == 'true'"
32-
Command="dotnet $(OutputPath)$(TargetFileName) generator crds --out $(KubeOpsCrdDir) --format $(KubeOpsCrdFormat) --use-old-crds"/>
33-
</Target>
34-
35-
<Target Name="GenerateRbac">
3626
<PropertyGroup>
3727
<!-- Configuration for the rbac generation -->
3828
<KubeOpsRbacDir Condition="'$(KubeOpsRbacDir)' == ''">$(KubeOpsConfigRoot)\rbac</KubeOpsRbacDir>
3929
<KubeOpsRbacFormat Condition="'$(KubeOpsRbacFormat)' == ''">Yaml</KubeOpsRbacFormat>
4030
</PropertyGroup>
4131

42-
<Message Text="Generating Rbac roles" Importance="high"/>
43-
<Message Text="Configuration path: $(KubeOpsRbacDir)" Importance="normal"/>
44-
45-
<Exec
46-
Command="dotnet $(OutputPath)$(TargetFileName) generator rbac --out $(KubeOpsRbacDir) --format $(KubeOpsRbacFormat)"/>
47-
</Target>
48-
49-
<Target Name="GenerateOperator">
5032
<PropertyGroup>
5133
<!-- Configuration for the operator manifest generation -->
5234
<KubeOpsOperatorDir Condition="'$(KubeOpsOperatorDir)' == ''">$(KubeOpsConfigRoot)\operator</KubeOpsOperatorDir>
5335
<KubeOpsOperatorFormat Condition="'$(KubeOpsOperatorFormat)' == ''">Yaml</KubeOpsOperatorFormat>
5436
</PropertyGroup>
5537

56-
<Message Text="Generating Operator yamls" Importance="high"/>
57-
<Message Text="Configuration path: $(KubeOpsOperatorDir)" Importance="normal"/>
58-
59-
<Exec
60-
Command="dotnet $(OutputPath)$(TargetFileName) generator operator --out $(KubeOpsOperatorDir) --format $(KubeOpsOperatorFormat)"/>
61-
</Target>
62-
63-
<Target Name="GenerateInstaller">
6438
<PropertyGroup>
6539
<!-- Configuration for the installer manifest generation -->
6640
<KubeOpsInstallerDir Condition="'$(KubeOpsInstallerDir)' == ''">$(KubeOpsConfigRoot)\install</KubeOpsInstallerDir>
6741
<KubeOpsInstallerFormat Condition="'$(KubeOpsInstallerFormat)' == ''">Yaml</KubeOpsInstallerFormat>
6842
</PropertyGroup>
43+
</Target>
6944

70-
<Message Text="Generating Installer yamls" Importance="high"/>
71-
<Message Text="Configuration path: $(KubeOpsInstallerDir)" Importance="normal"/>
45+
<Target Name="GenerateDockerfile" DependsOnTargets="BaseConfig">
46+
<Message Text="Generating Dockerfile" Importance="high" />
47+
<Message Text="Dockerfile path: $(KubeOpsDockerfilePath)" Importance="normal" />
7248

73-
<Message Condition="Exists('$(KubeOpsInstallerDir)')" Text="Installer dir exists, don't overwrite contents."
74-
Importance="high"/>
75-
<Exec Condition="!Exists('$(KubeOpsInstallerDir)')"
76-
Command="dotnet $(OutputPath)$(TargetFileName) generator installer --out $(KubeOpsInstallerDir) --format $(KubeOpsInstallerFormat) --crds-dir $(KubeOpsCrdDir) --rbac-dir $(KubeOpsRbacDir) --operator-dir $(KubeOpsOperatorDir)"/>
49+
<Message Condition="Exists('$(KubeOpsDockerfilePath)')" Text="Dockerfile already exists. Don't overwrite."
50+
Importance="high" />
51+
<Exec Condition="!Exists('$(KubeOpsDockerfilePath)')"
52+
Command="dotnet $(OutputPath)$(TargetFileName) generator docker --out $(KubeOpsDockerfilePath) --dotnet-tag $(KubeOpsDockerTag) --solution-dir $(SolutionDir) --target-file $(TargetFileName) --project-path $(ProjectPath)" />
7753
</Target>
7854

79-
<Target Name="GenerateAfterBuild" AfterTargets="Build">
80-
<PropertyGroup>
81-
<!-- Configuration for the base -->
82-
<KubeOpsBasePath Condition="'$(KubeOpsBasePath)' == ''">$(MSBuildProjectDirectory)</KubeOpsBasePath>
83-
</PropertyGroup>
55+
<Target Name="GenerateCrds" DependsOnTargets="BaseConfig">
56+
<Message Text="Generating CRDs" Importance="high" />
57+
<Message Text="Configuration path: $(KubeOpsCrdDir)" Importance="normal" />
8458

85-
<PropertyGroup>
86-
<!-- Configuration for the pathes where to store the generated yamls and elements -->
87-
<KubeOpsConfigRoot Condition="'$(KubeOpsConfigRoot)' == ''">$(KubeOpsBasePath)\config</KubeOpsConfigRoot>
88-
</PropertyGroup>
59+
<Exec Condition="'$(KubeOpsCrdUseOldCrds)' == 'false'"
60+
Command="dotnet $(OutputPath)$(TargetFileName) generator crds --out $(KubeOpsCrdDir) --format $(KubeOpsCrdFormat)" />
61+
<Exec Condition="'$(KubeOpsCrdUseOldCrds)' == 'true'"
62+
Command="dotnet $(OutputPath)$(TargetFileName) generator crds --out $(KubeOpsCrdDir) --format $(KubeOpsCrdFormat) --use-old-crds" />
63+
</Target>
64+
65+
<Target Name="GenerateRbac" DependsOnTargets="BaseConfig">
66+
<Message Text="Generating Rbac roles" Importance="high" />
67+
<Message Text="Configuration path: $(KubeOpsRbacDir)" Importance="normal" />
68+
69+
<Exec
70+
Command="dotnet $(OutputPath)$(TargetFileName) generator rbac --out $(KubeOpsRbacDir) --format $(KubeOpsRbacFormat)" />
71+
</Target>
72+
73+
<Target Name="GenerateOperator" DependsOnTargets="BaseConfig">
74+
<Message Text="Generating Operator yamls" Importance="high" />
75+
<Message Text="Configuration path: $(KubeOpsOperatorDir)" Importance="normal" />
76+
77+
<Exec
78+
Command="dotnet $(OutputPath)$(TargetFileName) generator operator --out $(KubeOpsOperatorDir) --format $(KubeOpsOperatorFormat)" />
79+
</Target>
80+
81+
<Target Name="GenerateInstaller" DependsOnTargets="BaseConfig">
82+
<Message Text="Generating Installer yamls" Importance="high" />
83+
<Message Text="Configuration path: $(KubeOpsInstallerDir)" Importance="normal" />
84+
85+
<Message Condition="Exists('$(KubeOpsInstallerDir)')" Text="Installer dir exists, don't overwrite contents."
86+
Importance="high" />
87+
<Exec Condition="!Exists('$(KubeOpsInstallerDir)')"
88+
Command="dotnet $(OutputPath)$(TargetFileName) generator installer --out $(KubeOpsInstallerDir) --format $(KubeOpsInstallerFormat) --crds-dir $(KubeOpsCrdDir) --rbac-dir $(KubeOpsRbacDir) --operator-dir $(KubeOpsOperatorDir)" />
89+
</Target>
8990

90-
<CallTarget Condition="'$(KubeOpsSkipDockerfile)' == ''" Targets="GenerateDockerfile"/>
91-
<CallTarget Condition="'$(KubeOpsSkipCrds)' == ''" Targets="GenerateCrds"/>
92-
<CallTarget Condition="'$(KubeOpsSkipRbac)' == ''" Targets="GenerateRbac"/>
93-
<CallTarget Condition="'$(KubeOpsSkipOperator)' == ''" Targets="GenerateOperator"/>
94-
<CallTarget Condition="'$(KubeOpsSkipInstaller)' == ''" Targets="GenerateInstaller"/>
91+
<Target Name="GenerateAfterBuild" AfterTargets="Build" DependsOnTargets="BaseConfig">
92+
<CallTarget Condition="'$(KubeOpsSkipDockerfile)' == ''" Targets="GenerateDockerfile" />
93+
<CallTarget Condition="'$(KubeOpsSkipCrds)' == ''" Targets="GenerateCrds" />
94+
<CallTarget Condition="'$(KubeOpsSkipRbac)' == ''" Targets="GenerateRbac" />
95+
<CallTarget Condition="'$(KubeOpsSkipOperator)' == ''" Targets="GenerateOperator" />
96+
<CallTarget Condition="'$(KubeOpsSkipInstaller)' == ''" Targets="GenerateInstaller" />
9597
</Target>
96-
</Project>
98+
</Project>

‎src/KubeOps/Operator/ApplicationBuilderExtensions.cs‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
1-
using KubeOps.Operator.Builder;
1+
using System;
2+
using System.Collections.Generic;
3+
using System.Linq;
4+
using System.Reflection;
5+
using KubeOps.Operator.Builder;
6+
using KubeOps.Operator.Webhooks;
27
using Microsoft.AspNetCore.Builder;
38
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
49
using Microsoft.Extensions.DependencyInjection;
10+
using Microsoft.Extensions.Logging;
511
using Prometheus;
612

713
namespace KubeOps.Operator
@@ -34,6 +40,37 @@ public static void UseKubernetesOperator(
3440
new HealthCheckOptions { Predicate = reg => reg.Tags.Contains(OperatorBuilder.ReadinessTag) });
3541

3642
endpoints.MapMetrics(settings.MetricsEndpoint);
43+
44+
var logger = app.ApplicationServices
45+
.GetRequiredService<ILoggerFactory>()
46+
.CreateLogger("ApplicationStartup");
47+
48+
foreach (var validator in app.ApplicationServices
49+
.GetService<IEnumerable<IValidationWebhook>>() ??
50+
new List<IValidationWebhook>())
51+
{
52+
var hookType = validator
53+
.GetType()
54+
.GetInterfaces()
55+
.FirstOrDefault(
56+
t => t.IsGenericType &&
57+
typeof(IValidationWebhook<>).IsAssignableFrom(t.GetGenericTypeDefinition()));
58+
if (hookType == null)
59+
{
60+
throw new Exception(
61+
$@"Validator ""{validator.GetType().Name}"" is not of IValidationWebhook<TEntity> type.");
62+
}
63+
64+
var registerMethod = hookType
65+
.GetMethods(BindingFlags.Instance | BindingFlags.NonPublic)
66+
.First(m => m.Name == "Register");
67+
68+
registerMethod.Invoke(validator, new object[] { endpoints });
69+
logger.LogInformation(
70+
@"Registered validation webhook ""{namespace}.{name}"".",
71+
validator.GetType().Namespace,
72+
validator.GetType().Name);
73+
}
3774
});
3875
}
3976
}

0 commit comments

Comments
 (0)