Skip to content

Commit 7f8837f

Browse files
madmecodesjuliusvonkohout
madmecodes
and
juliusvonkohout
authored
Document secure Kserve authentication via automated tests (#3056)
* update the kserve tests Signed-off-by: juliusvonkohout <[email protected]> * Secure KServe endpoints with oauth2-proxy authentication This change applies oauth2-proxy authentication to the cluster-local-gateway, ensuring KServe inference endpoints require proper authentication. Also adds a predictor-specific AuthorizationPolicy for test workflows. Fixes #2811 Signed-off-by: madmecodes <[email protected]> * Fix KServe workflows: use consistent paths, namespace handling, and add wait steps Signed-off-by: madmecodes <[email protected]> * Fix Fix KServe auth workflow by ordering components correctly Signed-off-by: madmecodes <[email protected]> * Fix order of KF Profile creation after multi-tenancy installation The Profile CRD needs to be installed via multi-tenancy components before attempting to create a user profile. This ensures the kubeflow-user-example-com namespace is properly created for tests. Signed-off-by: madmecodes <[email protected]> * test: namespace manual creation Update kserve_m2m_test.yaml workflow Signed-off-by: madmecodes <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update requirements.txt Signed-off-by: Julius von Kohout <[email protected]> * update: attempt to Enable secure KServe inferencing with oauth2-proxy authentication Signed-off-by: madmecodes <[email protected]> * enable istio-cni Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update dex_oauth2-proxy_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Delete tests/gh-actions/deploy-dex-login-environment/kustomization.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update dex_oauth2-proxy_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update dex_oauth2-proxy_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update dex_oauth2-proxy_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update centraldashboard_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update: Istio-cni-1-24 authorizationpolicy to use custom oauth2-proxy in cluster-local-gateway Signed-off-by: madmecodes <[email protected]> * Update: kserve_m2m_test.yaml attacker namespace test Signed-off-by: madmecodes <[email protected]> * Update: KServe AuthorizationPolicy Signed-off-by: madmecodes <[email protected]> * Update: using old KServe AuthorizationPolicy Signed-off-by: madmecodes <[email protected]> * Update: Kserve Auth policy namespace access Signed-off-by: madmecodes <[email protected]> * Update: fix the label in Auth Policy Signed-off-by: madmecodes <[email protected]> * Update: test, kserve_m2m_test using principals not namespaces Signed-off-by: madmecodes <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> * Update kserve_m2m_test.yaml Signed-off-by: Julius von Kohout <[email protected]> --------- Signed-off-by: juliusvonkohout <[email protected]> Signed-off-by: madmecodes <[email protected]> Signed-off-by: Julius von Kohout <[email protected]> Co-authored-by: juliusvonkohout <[email protected]>
1 parent 5b20e21 commit 7f8837f

File tree

8 files changed

+232
-100
lines changed

8 files changed

+232
-100
lines changed

‎.github/workflows/centraldashboard_test.yaml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ on:
55
- tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh
66
- .github/workflows/centraldashboard_test.yaml
77
- apps/centraldashboard/upstream/**
8-
- tests/gh-actions/install_istio.sh
8+
- tests/gh-actions/install_istio*.sh
99
- common/istio*/**
1010

1111
jobs:
@@ -21,9 +21,10 @@ jobs:
2121
- name: Install Istio
2222
run: ./tests/gh-actions/install_istio.sh
2323

24-
- name: Build & Apply manifests
24+
- name: Create kubeflow namespace
25+
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
26+
27+
- name: Install central-dashboard
2528
run: |
26-
cd apps/centraldashboard/upstream
27-
kubectl create ns kubeflow
28-
kustomize build overlays/kserve | kubectl apply -f -
29+
kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
2930
kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s

‎.github/workflows/dex_oauth2-proxy_test.yaml‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ on:
99
- common/istio*/**
1010
- experimental/security/PSS/*
1111
- common/dex/base/**
12-
- tests/gh-actions/install_istio.sh
12+
- tests/gh-actions/install_istio*.sh
1313

1414
jobs:
1515
build:
@@ -47,11 +47,24 @@ jobs:
4747
echo "Waiting for pods in auth namespace to become ready..."
4848
kubectl wait --for=condition=ready pods --all --timeout=180s -n auth
4949
50-
- name: Build & Apply manifests
50+
- name: Install central-dashboard
5151
run: |
52-
while ! kustomize build ./tests/gh-actions/deploy-dex-login-environment | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done
52+
kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
5353
kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s
5454
55+
- name: Create KF Profile
56+
run: |
57+
kustomize build common/user-namespace/base | kubectl apply -f -
58+
sleep 30 # for the Profile controller to create the namespace from the profile
59+
PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
60+
if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
61+
echo "Error: profiles-deployment pod not found in kubeflow namespace."
62+
exit 1
63+
fi
64+
kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
65+
KF_PROFILE=kubeflow-user-example-com
66+
kubectl -n $KF_PROFILE get pods,configmaps,secrets
67+
5568
- name: port forward
5669
run: |
5770
ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')

‎.github/workflows/kserve_m2m_test.yaml‎

Lines changed: 196 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,14 @@ on:
77
- apps/kserve/**
88
- tests/gh-actions/install_kserve.sh
99
- common/istio*/**
10-
- tests/gh-actions/install_istio.sh
10+
- tests/gh-actions/install_istio*.sh
1111
- common/oauth2-proxy/**
1212
- tests/gh-actions/install_oauth2-proxy.sh
1313
- common/cert-manager/**
1414
- tests/gh-actions/install_cert_manager.sh
1515
- common/knative/**
1616
- tests/gh-actions/install_knative.sh
1717

18-
1918
jobs:
2019
build:
2120
runs-on: ubuntu-latest
@@ -29,26 +28,128 @@ jobs:
2928
- name: Install kubectl
3029
run: ./tests/gh-actions/install_kubectl.sh
3130

32-
- name: Create kubeflow namespace
33-
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
34-
35-
- name: Install Istio
36-
run: ./tests/gh-actions/install_istio.sh
31+
- name: Install Istio CNI
32+
run: ./tests/gh-actions/install_istio-cni.sh
3733

3834
- name: Install oauth2-proxy
3935
run: ./tests/gh-actions/install_oauth2-proxy.sh
4036

4137
- name: Install cert-manager
4238
run: ./tests/gh-actions/install_cert_manager.sh
4339

44-
- name: Install knative
45-
run: ./tests/gh-actions/install_knative.sh
40+
- name: Create kubeflow namespace
41+
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
42+
43+
- name: Install knative CNI
44+
run: ./tests/gh-actions/install_knative-cni.sh
4645

4746
- name: Install KServe
4847
run: ./tests/gh-actions/install_kserve.sh
4948

50-
- name: Create test namespace # TODO to be removed and instead we shall use kubeflow-user-example-com
51-
run: kubectl create ns kserve-test
49+
- name: Install KF Multi Tenancy
50+
run: ./tests/gh-actions/install_multi_tenancy.sh
51+
52+
- name: Install kubeflow-istio-resources
53+
run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
54+
55+
- name: Create KF Profile
56+
run: |
57+
kustomize build common/user-namespace/base | kubectl apply -f -
58+
sleep 30 # for the Profile controller to create the namespace from the profile
59+
60+
PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
61+
if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
62+
echo "Error: profiles-deployment pod not found in kubeflow namespace."
63+
exit 1
64+
fi
65+
kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
66+
KF_PROFILE=kubeflow-user-example-com
67+
kubectl -n $KF_PROFILE get pods,configmaps,secrets
68+
69+
- name: Diagnose KServe Service Labels
70+
run: |
71+
echo "=== KServe Predictor Service Labels ==="
72+
kubectl get pods -n kubeflow-user-example-com -l serving.knative.dev/service=isvc-sklearn-predictor-default --show-labels
73+
74+
# TODO for follow up PR
75+
#- name: Apply KServe predictor AuthorizationPolicy
76+
# run: |
77+
# cat <<EOF | kubectl apply -f -
78+
# apiVersion: security.istio.io/v1beta1
79+
# kind: AuthorizationPolicy
80+
# metadata:
81+
# name: sklearn-iris-predictor-allow
82+
# namespace: kubeflow-user-example-com
83+
# spec:
84+
# selector:
85+
# matchLabels:
86+
# serving.knative.dev/service: isvc-sklearn-predictor
87+
# action: ALLOW
88+
# rules:
89+
# - from:
90+
# - source:
91+
# namespaces:
92+
# - "istio-system"
93+
# - "knative-serving"
94+
# - "kubeflow"
95+
# - "kubeflow-user-example-com"
96+
# - principals:
97+
# - "cluster.local/ns/kubeflow-user-example-com/sa/default-editor"
98+
# - "cluster.local/ns/kubeflow-user-example-com/sa/default"
99+
# - "cluster.local/ns/kubeflow-user-example-com/sa/default-viewer"
100+
# to:
101+
# - operation:
102+
# paths:
103+
# - "/v1/models/*"
104+
# - "/v2/models/*"
105+
# EOF
106+
107+
- name: Apply INSECURE KServe AuthorizationPolicy
108+
run: |
109+
cat <<EOF | kubectl apply -f -
110+
apiVersion: security.istio.io/v1beta1
111+
kind: AuthorizationPolicy
112+
metadata:
113+
name: allow-in-cluster-kserve
114+
namespace: kubeflow-user-example-com
115+
spec:
116+
rules:
117+
- to:
118+
- operation:
119+
paths:
120+
- /v1/models/*
121+
- /v2/models/*
122+
EOF
123+
124+
- name: Add KServe path-based routing for external access
125+
run: |
126+
cat <<EOF | kubectl apply -f -
127+
apiVersion: networking.istio.io/v1beta1
128+
kind: VirtualService
129+
metadata:
130+
name: isvc-sklearn-external
131+
namespace: kubeflow-user-example-com
132+
spec:
133+
gateways:
134+
- kubeflow/kubeflow-gateway
135+
hosts:
136+
- '*'
137+
http:
138+
- match:
139+
- uri:
140+
prefix: /kserve/kubeflow-user-example-com/isvc-sklearn/
141+
rewrite:
142+
uri: /
143+
route:
144+
- destination:
145+
host: knative-local-gateway.istio-system.svc.cluster.local
146+
headers:
147+
request:
148+
set:
149+
Host: isvc-sklearn-predictor-default.kubeflow-user-example-com.svc.cluster.local
150+
weight: 100
151+
timeout: 300s
152+
EOF
52153
53154
- name: Setup python 3.12
54155
uses: actions/setup-python@v4
@@ -64,26 +165,97 @@ jobs:
64165
nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 &
65166
while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready
66167
67-
- name: Run kserve tests with m2m token from SA default/default # TODO Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
168+
- name: Verify CNI is being used instead of init containers
169+
run: |
170+
INIT_CONTAINERS=$(kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.initContainers[*]}{.name}{"\n"}{end}{"\n"}{end}' | grep istio-init || echo "none")
171+
if [[ "$INIT_CONTAINERS" != "none" ]]; then
172+
echo "Error: Found istio-init containers, CNI is not being used properly"
173+
echo "$INIT_CONTAINERS"
174+
exit 1
175+
else
176+
echo "Success: No istio-init containers found, CNI is working correctly"
177+
fi
178+
179+
- name: Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
68180
run: |
69-
# TODO run the tests against kubeflow-user-example-com
70181
export KSERVE_INGRESS_HOST_PORT=localhost:8080
71-
export KSERVE_M2M_TOKEN="$(kubectl -n default create token default)"
72-
# TODO export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
73-
# TODO in contrib/kserve/tests/utils.py use KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
182+
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
74183
cd ./apps/kserve/tests && pytest . -vs --log-level info
75184
76-
- name: Run and fail kserve tests without kserve m2m token
185+
- name: Detailed KServe Access Diagnostics
77186
run: |
78187
export KSERVE_INGRESS_HOST_PORT=localhost:8080
79-
cd ./apps/kserve/tests
80-
if pytest . -vs --log-level info; then
81-
echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
82-
else
83-
echo "Task failed successfully!"
84-
echo "This is a provisional way of testing that m2m is enabled for kserve."
85-
fi
188+
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
189+
190+
echo "=== AuthorizationPolicy Details ==="
191+
kubectl get authorizationpolicy -n kubeflow-user-example-com -o yaml
192+
193+
echo "=== Detailed Curl Test ==="
194+
curl -vv \
195+
-H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
196+
-H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
197+
-H "Content-Type: application/json" \
198+
"http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
199+
-d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
200+
201+
# TODO FOR FOLLOW UP PR
202+
#- name: Run and fail kserve tests without kserve m2m token
203+
#run: |
204+
# export KSERVE_INGRESS_HOST_PORT=localhost:8080
205+
# cd ./apps/kserve/tests
206+
# if pytest . -vs --log-level info; then
207+
# echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
208+
# else
209+
# echo "Task failed successfully!"
210+
# echo "This is a provisional way of testing that m2m is enabled for kserve."
211+
# fi
212+
213+
# TODO FOR FOLLOW UP PR
214+
#- name: Test that token from attacker namespace is rejected
215+
# run: |
216+
# export KSERVE_INGRESS_HOST_PORT=localhost:8080
217+
# kubectl create ns kubeflow-user-example-com-attacker
218+
# kubectl create serviceaccount attacker-sa -n kubeflow-user-example-com-attacker
219+
# export ATTACKER_TOKEN="$(kubectl -n kubeflow-user-example-com-attacker create token attacker-sa)"
220+
# RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
221+
# -H "Authorization: Bearer ${ATTACKER_TOKEN}" \
222+
# -H "Content-Type: application/json" \
223+
# "http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
224+
# -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}')
225+
# if [[ "$RESPONSE" == "403" || "$RESPONSE" == "401" ]]; then
226+
# echo "Security test passed: Request with attacker token was correctly rejected with $RESPONSE"
227+
# else
228+
# echo "Security test failed: Request with attacker token returned $RESPONSE instead of 403/401"
229+
# exit 1
230+
# fi
231+
232+
- name: Test path-based external access
233+
run: |
234+
export KSERVE_INGRESS_HOST_PORT=localhost:8080
235+
export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
236+
237+
# Test external path-based access
238+
curl -v -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
239+
-H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
240+
-H "Content-Type: application/json" \
241+
"http://${KSERVE_INGRESS_HOST_PORT}/kserve/kubeflow-user-example-com/isvc-sklearn/v1/models/isvc-sklearn:predict" \
242+
-d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
86243
87244
- name: Run kserve models webapp test
88245
run: |
89246
kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app
247+
248+
- name: Apply Pod Security Standards baseline levels
249+
run: ./tests/gh-actions/enable_baseline_PSS.sh
250+
251+
- name: Unapply applied baseline labels
252+
run: |
253+
NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow" "knative-serving")
254+
for NAMESPACE in "${NAMESPACES[@]}"; do
255+
if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
256+
kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
257+
fi
258+
done
259+
260+
- name: Applying Pod Security Standards restricted levels
261+
run: ./tests/gh-actions/enable_restricted_PSS.sh

‎apps/kserve/tests/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
pytest>=7.0.0
2-
kserve>=0.12.1
2+
kserve>=0.14.1
33
kubernetes>=18.20.0
44
requests>=2.18.4

‎apps/kserve/tests/utils.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
logging.basicConfig(level=logging.INFO)
2727

2828
KSERVE_NAMESPACE = "kserve"
29-
KSERVE_TEST_NAMESPACE = "kserve-test"
29+
KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
3030
MODEL_CLASS_NAME = "modelClass"
3131

3232

‎common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
1-
# Allow all traffic to the cluster-local-gateway
1+
# Enforce OAuth2-proxy authentication for cluster-local-gateway
22
apiVersion: security.istio.io/v1beta1
33
kind: AuthorizationPolicy
44
metadata:
5-
name: cluster-local-gateway
5+
name: cluster-local-gateway-oauth2-proxy
66
spec:
7-
action: ALLOW
7+
action: CUSTOM
8+
provider:
9+
name: oauth2-proxy
810
selector:
911
# Same as the cluster-local-gateway Service selector
1012
matchLabels:
1113
app: cluster-local-gateway
1214
istio: cluster-local-gateway
1315
rules:
14-
- {}
16+
- {}

‎common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
1-
# Allow all traffic to the cluster-local-gateway
1+
# Enforce OAuth2-proxy authentication for cluster-local-gateway
22
apiVersion: security.istio.io/v1beta1
33
kind: AuthorizationPolicy
44
metadata:
5-
name: cluster-local-gateway
5+
name: cluster-local-gateway-oauth2-proxy
66
spec:
7-
action: ALLOW
7+
action: CUSTOM
8+
provider:
9+
name: oauth2-proxy
810
selector:
911
# Same as the cluster-local-gateway Service selector
1012
matchLabels:
1113
app: cluster-local-gateway
1214
istio: cluster-local-gateway
1315
rules:
14-
- {}
16+
- {}

0 commit comments

Comments
 (0)