Skip to content

Commit b94c592

Browse files
authored
feat: Use Localtunnel to improve developer experience (#259)
This closes #253. When developing webhooks, one needed to start the operator locally, then use some technology like "localtunnel" or "ngrok" to have a tunnel (with HTTPS) to the local running operator and then register that given randomized url to Kubernetes to actually call the migrator/validator. This pull request uses localtunnel as a libary by adding AddWebhookLocaltunnel to the operator builder. With this, the operator creates a localtunnel during startup and registers itself within Kubernetes with the returned url. Note: this should never be used in production.
1 parent 671579b commit b94c592

18 files changed

+360
-72
lines changed

docs/docs/webhooks.md

Lines changed: 63 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Webhooks
22

33
Kubernetes supports various webhooks to extend the normal api behaviour
4-
of the master api. Those are documented on the
4+
of the master api. Those are documented on the
55
[kubernetes website](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/).
66

77
`KubeOps` supports the following webhooks out of the box:
8+
89
- Validator / Validation
910
- Mutator / Mutation
1011

@@ -24,6 +25,7 @@ generate a CA certificate for you.
2425

2526
So if you add a webhook to your operator the following changes
2627
to the normal deployment of the operator will happen:
28+
2729
1. During "after build" phase, the sdk will generate
2830
a CA-certificate for self signed certificates for you.
2931
2. The ca certificate and the corresponding key are added
@@ -61,9 +63,64 @@ trigger a POST call to the operator.
6163
6264
## Local development
6365

64-
It is possible to test webhooks locally. For this, you need
65-
to register the webhook via dependency injection with the corresponding
66-
method (in the builder) and then start your operator.
66+
It is possible to test / debug webhooks locally. For this, you need
67+
to implement the webhook and use assembly-scanning (or the
68+
operator builder if you disabled scanning) to register
69+
the webhook type.
70+
71+
There are two possibilities to tell Kubernetes that it should
72+
call your local running operator for the webhooks. The url
73+
that Kubernetes addresses _must_ be an HTTPS address.
74+
75+
### Using `AddWebhookLocaltunnel`
76+
77+
In your `Startup.cs` you can use the `IOperatorBuilder`
78+
method `AddWebhookLocaltunnel` to add an automatic
79+
localtunnel instance to your operator.
80+
81+
This will cause the operator to register a hosted service that
82+
creates a tunnel and then registers itself to Kubernetes
83+
with the created proxy-url. Now all calls are automatically
84+
forwarded via HTTPS to your operator.
85+
86+
```csharp
87+
namespace KubeOps.TestOperator
88+
{
89+
public class Startup
90+
{
91+
public void ConfigureServices(IServiceCollection services)
92+
{
93+
services
94+
.AddKubernetesOperator()
95+
#if DEBUG
96+
.AddWebhookLocaltunnel()
97+
#endif
98+
;
99+
services.AddTransient<IManager, TestManager.TestManager>();
100+
}
101+
102+
public void Configure(IApplicationBuilder app)
103+
{
104+
app.UseKubernetesOperator();
105+
}
106+
}
107+
}
108+
```
109+
110+
> [!WARNING]
111+
> It is _strongly_ advices against using auto-webhooks
112+
> with localtunnel in production. This feature
113+
> is intended to improve the developer experience
114+
> while coding operators.
115+
116+
> [!NOTE]
117+
> Some IDEs (like Rider from JetBrains) do not correctly
118+
> terminate debugged applications. Hence, the
119+
> webhook registration remains in Kubernetes. If you remove
120+
> webhooks from your operator, you need to remove the
121+
> registration within Kubernetes as well.
122+
123+
### Using external proxy
67124

68125
The operator will run on a specific http address, depending on your
69126
configuration.
@@ -87,6 +144,7 @@ Webhooks are registered in a **scoped** manner to the DI system.
87144
They behave like asp.net api controller.
88145

89146
The implementation of a validator is fairly simple:
147+
90148
- Create a class somewhere in your project.
91149
- Implement the @"KubeOps.Operator.Webhooks.IValidationWebhook`1" interface.
92150
- Define the @"KubeOps.Operator.Webhooks.IAdmissionWebhook`2.Operations"
@@ -138,6 +196,7 @@ the object that is later passed to the validators and to the Kubernetes
138196
API.
139197

140198
The implementation of a mutator is fairly simple:
199+
141200
- Create a class somewhere in your project.
142201
- Implement the @"KubeOps.Operator.Webhooks.IMutationWebhook`1" interface.
143202
- Define the @"KubeOps.Operator.Webhooks.IAdmissionWebhook`2.Operations"

src/KubeOps/KubeOps.csproj

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
<PackageReference Include="CompareNETObjects" Version="4.73.0" />
2828
<PackageReference Include="DotnetKubernetesClient" Version="2.0.7" />
2929
<PackageReference Include="JsonDiffPatch" Version="2.0.55" />
30+
<PackageReference Include="Localtunnel" Version="1.0.3" />
3031
<PackageReference Include="McMaster.Extensions.CommandLineUtils" Version="3.1.0" />
3132
<PackageReference Include="McMaster.Extensions.Hosting.CommandLine" Version="3.1.0" />
3233
<PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks" Version="5.0.9" />

src/KubeOps/Operator/Builder/AssemblyScanner.cs

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,31 @@ public AssemblyScanner(IOperatorBuilder operatorBuilder)
2525
_registrationDefinitions =
2626
new (Type Type, string MethodName)[]
2727
{
28-
new() { Type = typeof(IKubernetesObject<V1ObjectMeta>), MethodName = nameof(OperatorBuilderExtensions.AddEntity) },
29-
new() { Type = typeof(IResourceController<>), MethodName = nameof(OperatorBuilderExtensions.AddController) },
30-
new() { Type = typeof(IResourceFinalizer<>), MethodName = nameof(OperatorBuilderExtensions.AddFinalizer) },
31-
new() { Type = typeof(IValidationWebhook<>), MethodName = nameof(OperatorBuilderExtensions.AddValidationWebhook) },
32-
new() { Type = typeof(IMutationWebhook<>), MethodName = nameof(OperatorBuilderExtensions.AddMutationWebhook) },
28+
new()
29+
{
30+
Type = typeof(IKubernetesObject<V1ObjectMeta>),
31+
MethodName = nameof(OperatorBuilderExtensions.AddEntity),
32+
},
33+
new()
34+
{
35+
Type = typeof(IResourceController<>),
36+
MethodName = nameof(OperatorBuilderExtensions.AddController),
37+
},
38+
new()
39+
{
40+
Type = typeof(IResourceFinalizer<>),
41+
MethodName = nameof(OperatorBuilderExtensions.AddFinalizer),
42+
},
43+
new()
44+
{
45+
Type = typeof(IValidationWebhook<>),
46+
MethodName = nameof(OperatorBuilderExtensions.AddValidationWebhook),
47+
},
48+
new()
49+
{
50+
Type = typeof(IMutationWebhook<>),
51+
MethodName = nameof(OperatorBuilderExtensions.AddMutationWebhook),
52+
},
3353
}
3454
.Select<(Type Type, string MethodName), (Type Type, MethodInfo RegistrationMethod)>(
3555
t => new()
@@ -55,8 +75,10 @@ public IAssemblyScanner AddAssembly(Assembly assembly)
5575
.Where(
5676
t => t.ComponentType.GetInterfaces()
5777
.Any(
58-
i => (i.IsConstructedGenericType && i.GetGenericTypeDefinition().IsEquivalentTo(t.RegistrationDefinition.Type)) ||
59-
(t.RegistrationDefinition.Type.IsConstructedGenericType && i.IsEquivalentTo(t.RegistrationDefinition.Type))))
78+
i => (i.IsConstructedGenericType &&
79+
i.GetGenericTypeDefinition().IsEquivalentTo(t.RegistrationDefinition.Type)) ||
80+
(t.RegistrationDefinition.Type.IsConstructedGenericType &&
81+
i.IsEquivalentTo(t.RegistrationDefinition.Type))))
6082
.Select(
6183
t => t.RegistrationDefinition.RegistrationMethod.MakeGenericMethod(t.ComponentType));
6284

src/KubeOps/Operator/Builder/ComponentRegistrar.cs

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,14 @@ internal class ComponentRegistrar : IComponentRegistrar
1919

2020
public ImmutableHashSet<EntityRegistration> EntityRegistrations => _entityRegistrations.ToImmutableHashSet();
2121

22-
public ImmutableHashSet<ControllerRegistration> ControllerRegistrations => _controllerRegistrations.ToImmutableHashSet();
22+
public ImmutableHashSet<ControllerRegistration> ControllerRegistrations =>
23+
_controllerRegistrations.ToImmutableHashSet();
2324

24-
public ImmutableHashSet<FinalizerRegistration> FinalizerRegistrations => _finalizerRegistrations.ToImmutableHashSet();
25+
public ImmutableHashSet<FinalizerRegistration> FinalizerRegistrations =>
26+
_finalizerRegistrations.ToImmutableHashSet();
2527

26-
public ImmutableHashSet<ValidatorRegistration> ValidatorRegistrations => _validatorRegistrations.ToImmutableHashSet();
28+
public ImmutableHashSet<ValidatorRegistration> ValidatorRegistrations =>
29+
_validatorRegistrations.ToImmutableHashSet();
2730

2831
public ImmutableHashSet<MutatorRegistration> MutatorRegistrations => _mutatorRegistrations.ToImmutableHashSet();
2932

src/KubeOps/Operator/Builder/IOperatorBuilder.cs

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,5 +140,37 @@ IOperatorBuilder AddValidationWebhook<TImplementation, TEntity>()
140140
IOperatorBuilder AddMutationWebhook<TImplementation, TEntity>()
141141
where TImplementation : class, IMutationWebhook<TEntity>
142142
where TEntity : IKubernetesObject<V1ObjectMeta>;
143+
144+
/// <summary>
145+
/// <para>
146+
/// Adds a hosted service to the system that creates a "localtunnel"
147+
/// (http://localtunnel.github.io/www/) to the running application.
148+
/// The tunnel points to the configured host/port configuration and then
149+
/// registers itself as webhook target within Kubernetes. This
150+
/// enables developers to easily create webhooks without the requirement
151+
/// of registering ngrok / localtunnel urls themselves.
152+
/// </para>
153+
/// <para>
154+
/// This is a convenience method to improve the developer experience.
155+
/// Since some IDEs do not gracefully shutdown applications that
156+
/// have a debugger attached, the registration may not be removed.
157+
/// </para>
158+
/// <para>
159+
/// It is strongly recommended to use this method only while developing
160+
/// or debugging an operator. *Never* use this in production.
161+
/// </para>
162+
/// </summary>
163+
/// <param name="hostname">The hostname that the tunnel should target to proxy.</param>
164+
/// <param name="port">The target port to proxy.</param>
165+
/// <param name="isHttps">If set to true, the target uses HTTPS.</param>
166+
/// <param name="allowUntrustedCertificates">
167+
/// If the target uses HTTPS, should self signed / untrusted certificates be allowed or not.
168+
/// </param>
169+
/// <returns>The builder for chaining.</returns>
170+
IOperatorBuilder AddWebhookLocaltunnel(
171+
string hostname = "localhost",
172+
short port = 5000,
173+
bool isHttps = false,
174+
bool allowUntrustedCertificates = true);
143175
}
144176
}

src/KubeOps/Operator/Builder/OperatorBuilder.cs

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
using Microsoft.Extensions.DependencyInjection;
1818
using Microsoft.Extensions.DependencyInjection.Extensions;
1919
using Microsoft.Extensions.Diagnostics.HealthChecks;
20+
using Microsoft.Extensions.Logging;
2021
using Prometheus;
2122
using YamlDotNet.Serialization;
2223

@@ -130,6 +131,29 @@ public IOperatorBuilder AddMutationWebhook<TImplementation, TEntity>()
130131
return this;
131132
}
132133

134+
public IOperatorBuilder AddWebhookLocaltunnel(
135+
string hostname = "localhost",
136+
short port = 5000,
137+
bool isHttps = false,
138+
bool allowUntrustedCertificates = true)
139+
{
140+
Services.AddHostedService(
141+
services => new WebhookLocalTunnel(
142+
services.GetRequiredService<ILogger<WebhookLocalTunnel>>(),
143+
services.GetRequiredService<OperatorSettings>(),
144+
services.GetRequiredService<IKubernetesClient>(),
145+
services.GetRequiredService<MutatingWebhookConfigurationBuilder>(),
146+
services.GetRequiredService<ValidatingWebhookConfigurationBuilder>())
147+
{
148+
Host = hostname,
149+
Port = port,
150+
IsHttps = isHttps,
151+
AllowUntrustedCertificates = allowUntrustedCertificates,
152+
});
153+
154+
return this;
155+
}
156+
133157
internal IOperatorBuilder AddOperatorBase(OperatorSettings settings)
134158
{
135159
if (settings.EnableAssemblyScanning)

src/KubeOps/Operator/Commands/Management/Webhooks/Register.cs

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,10 @@ public Register(
7878

7979
public async Task<int> OnExecuteAsync(CommandLineApplication app)
8080
{
81-
await app.Out.WriteLineAsync($"Found {_componentRegistrar.ValidatorRegistrations.Count} validator registrations.");
82-
await app.Out.WriteLineAsync($"Found {_componentRegistrar.MutatorRegistrations.Count} mutator registrations.");
81+
await app.Out.WriteLineAsync(
82+
$"Found {_componentRegistrar.ValidatorRegistrations.Count} validator registrations.");
83+
await app.Out.WriteLineAsync(
84+
$"Found {_componentRegistrar.MutatorRegistrations.Count} mutator registrations.");
8385

8486
var webhookConfig = new WebhookConfig(
8587
_settings.Name,

src/KubeOps/Operator/Controller/ManagedResourceController{TEntity}.cs

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -86,15 +86,17 @@ public ManagedResourceController(
8686
.Select(
8787
data => Observable.Return(data).Delay(data.Delay))
8888
.Switch()
89-
.Select(data =>
90-
Observable.FromAsync(async () =>
91-
{
92-
var queuedEvent = await UpdateResourceData(data.Resource);
93-
94-
return data.ResourceEvent.HasValue && queuedEvent != null
95-
? queuedEvent with { ResourceEvent = data.ResourceEvent.Value }
96-
: queuedEvent;
97-
}))
89+
.Select(
90+
data =>
91+
Observable.FromAsync(
92+
async () =>
93+
{
94+
var queuedEvent = await UpdateResourceData(data.Resource);
95+
96+
return data.ResourceEvent.HasValue && queuedEvent != null
97+
? queuedEvent with { ResourceEvent = data.ResourceEvent.Value }
98+
: queuedEvent;
99+
}))
98100
.Switch()
99101
.Where(data => data != null)
100102
.Do(
@@ -338,7 +340,6 @@ await scope.ServiceProvider.GetRequiredService<IFinalizerManager<TEntity>>()
338340
resource.Name(),
339341
retryCount + 1);
340342
_erroredEvents.OnNext(data with { RetryCount = retryCount + 1 });
341-
return;
342343
}
343344
}
344345

src/KubeOps/Operator/OperatorBuilderExtensions.cs

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@ public static class OperatorBuilderExtensions
2626
public static IOperatorBuilder AddController<TImplementation>(this IOperatorBuilder builder)
2727
where TImplementation : class
2828
{
29-
var entityTypes = typeof(TImplementation).GetInterfaces().Where(t =>
30-
t.IsConstructedGenericType &&
31-
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IResourceController<>)))
29+
var entityTypes = typeof(TImplementation).GetInterfaces()
30+
.Where(
31+
t =>
32+
t.IsConstructedGenericType &&
33+
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IResourceController<>)))
3234
.Select(i => i.GenericTypeArguments[0]);
3335

3436
var genericRegistrationMethod = builder
@@ -61,9 +63,11 @@ public static IOperatorBuilder AddController<TImplementation>(this IOperatorBuil
6163
public static IOperatorBuilder AddFinalizer<TImplementation>(this IOperatorBuilder builder)
6264
where TImplementation : class
6365
{
64-
var entityTypes = typeof(TImplementation).GetInterfaces().Where(t =>
65-
t.IsConstructedGenericType &&
66-
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IResourceFinalizer<>)))
66+
var entityTypes = typeof(TImplementation).GetInterfaces()
67+
.Where(
68+
t =>
69+
t.IsConstructedGenericType &&
70+
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IResourceFinalizer<>)))
6771
.Select(i => i.GenericTypeArguments[0]);
6872

6973
var genericRegistrationMethod = builder
@@ -96,9 +100,11 @@ public static IOperatorBuilder AddFinalizer<TImplementation>(this IOperatorBuild
96100
public static IOperatorBuilder AddValidationWebhook<TImplementation>(this IOperatorBuilder builder)
97101
where TImplementation : class
98102
{
99-
var entityTypes = typeof(TImplementation).GetInterfaces().Where(t =>
100-
t.IsConstructedGenericType &&
101-
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IValidationWebhook<>)))
103+
var entityTypes = typeof(TImplementation).GetInterfaces()
104+
.Where(
105+
t =>
106+
t.IsConstructedGenericType &&
107+
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IValidationWebhook<>)))
102108
.Select(i => i.GenericTypeArguments[0]);
103109

104110
var genericRegistrationMethod = builder
@@ -131,9 +137,11 @@ public static IOperatorBuilder AddValidationWebhook<TImplementation>(this IOpera
131137
public static IOperatorBuilder AddMutationWebhook<TImplementation>(this IOperatorBuilder builder)
132138
where TImplementation : class
133139
{
134-
var entityTypes = typeof(TImplementation).GetInterfaces().Where(t =>
135-
t.IsConstructedGenericType &&
136-
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IMutationWebhook<>)))
140+
var entityTypes = typeof(TImplementation).GetInterfaces()
141+
.Where(
142+
t =>
143+
t.IsConstructedGenericType &&
144+
t.GetGenericTypeDefinition().IsEquivalentTo(typeof(IMutationWebhook<>)))
137145
.Select(i => i.GenericTypeArguments[0]);
138146

139147
var genericRegistrationMethod = builder

src/KubeOps/Operator/Rbac/RbacBuilder.cs

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,17 @@ internal class RbacBuilder : IRbacBuilder
1616
public RbacBuilder(IComponentRegistrar componentRegistrar)
1717
{
1818
var controllerTypes = componentRegistrar.ControllerRegistrations
19-
.Select(t => t.ControllerType).ToList();
19+
.Select(t => t.ControllerType)
20+
.ToList();
2021
var finalizerTypes = componentRegistrar.FinalizerRegistrations
21-
.Select(t => t.FinalizerType).ToList();
22+
.Select(t => t.FinalizerType)
23+
.ToList();
2224
var validatorTypes = componentRegistrar.ValidatorRegistrations
23-
.Select(t => t.ValidatorType).ToList();
25+
.Select(t => t.ValidatorType)
26+
.ToList();
2427
var mutatorTypes = componentRegistrar.MutatorRegistrations
25-
.Select(t => t.MutatorType).ToList();
28+
.Select(t => t.MutatorType)
29+
.ToList();
2630

2731
_componentTypes = Enumerable.Empty<Type>()
2832
.Concat(controllerTypes)

0 commit comments

Comments
 (0)