Skip to content

Commit 5f33881

Browse files
madmecodesmadmecodes
authored
Add VirtualService conflict troubleshooting for KServe path-based Routing (#3247)
* docs: Add VirtualService conflict troubleshooting for KServe path-based routing Signed-off-by: madmecodes <[email protected]> * update: merged kserve tests into one file Signed-off-by: madmecodes <[email protected]> --------- Signed-off-by: madmecodes <[email protected]>
1 parent 6aad970 commit 5f33881

File tree

5 files changed

+201
-174
lines changed

5 files changed

+201
-174
lines changed

‎.github/workflows/kserve_jwt_authentication_test.yaml‎

Lines changed: 0 additions & 88 deletions
This file was deleted.

‎.github/workflows/kserve_secure_test.yaml‎

Lines changed: 0 additions & 83 deletions
This file was deleted.

‎.github/workflows/kserve_test.yaml‎

Lines changed: 143 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,25 +3,29 @@ on:
33
pull_request:
44
paths:
55
- tests/install_KinD_create_KinD_cluster_install_kustomize.sh
6-
- .github/workflows/kserve_m2m_test.yaml
6+
- .github/workflows/kserve_test.yaml
77
- applications/kserve/**
8+
- apps/kserve/**
89
- tests/kserve/**
910
- tests/kserve_test.sh
1011
- tests/kserve_install.sh
12+
- tests/kserve_jwt_authentication_test.sh
1113
- common/istio*/**
1214
- common/oauth2-proxy/**
1315
- tests/oauth2-proxy_install.sh
1416
- common/cert-manager/**
1517
- tests/istio*
1618
- common/knative/**
1719
- tests/knative_install.sh
20+
- tests/*authentication*test.sh
21+
- tests/final_validation.sh
1822

1923
permissions:
2024
contents: read
2125
actions: read
2226

2327
jobs:
24-
build:
28+
test-basic-kserve:
2529
runs-on: ubuntu-latest
2630
steps:
2731
- name: Checkout
@@ -93,3 +97,140 @@ jobs:
9397
9498
- name: Apply Pod Security Standards restricted levels
9599
run: ./tests/PSS_enable.sh
100+
101+
test-jwt-authentication:
102+
runs-on: ubuntu-latest
103+
steps:
104+
- name: Checkout
105+
uses: actions/checkout@v4
106+
107+
- name: Install KinD, Create KinD cluster and Install kustomize
108+
run: ./tests/install_KinD_create_KinD_cluster_install_kustomize.sh
109+
110+
- name: Install kubectl
111+
run: ./tests/kubectl_install.sh
112+
113+
- name: Create kubeflow namespace
114+
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
115+
116+
- name: Install Istio CNI
117+
run: ./tests/istio-cni_install.sh
118+
119+
- name: Install oauth2-proxy
120+
run: ./tests/oauth2-proxy_install.sh
121+
122+
- name: Install knative CNI with secure cluster-local-gateway
123+
run: ./tests/knative_install.sh
124+
125+
- name: Verify secure cluster-local-gateway configuration
126+
run: |
127+
kubectl get authorizationpolicy,requestauthentication -n istio-system | grep cluster-local-gateway
128+
kubectl get requestauthentication cluster-local-gateway-jwt -n istio-system -o yaml
129+
kubectl get authorizationpolicy cluster-local-gateway -n istio-system -o yaml
130+
kubectl get authorizationpolicy cluster-local-gateway-require-jwt -n istio-system -o yaml
131+
132+
- name: Setup python 3.12
133+
uses: actions/setup-python@v4
134+
with:
135+
python-version: 3.12
136+
137+
- name: Port forward
138+
run: ./tests/port_forward_gateway.sh
139+
140+
- name: Wait for cluster-local-gateway to be ready
141+
run: |
142+
kubectl wait --for=condition=Available --timeout=120s deployment/cluster-local-gateway -n istio-system
143+
sleep 100
144+
145+
- name: Run Basic JWT Authentication Tests
146+
run: |
147+
export KSERVE_INGRESS_HOST_PORT=localhost:8080
148+
curl -s -o /dev/null -w "%{http_code}" -H "Host: test.example.com" "http://localhost:8080/" | grep -q "403"
149+
150+
- name: Run Knative Service JWT Authentication Tests
151+
run: |
152+
export KSERVE_INGRESS_HOST_PORT=localhost:8080
153+
./tests/knative_authentication_test.sh
154+
155+
- name: Test External Access Configuration
156+
run: |
157+
export KSERVE_INGRESS_HOST_PORT=localhost:8080
158+
./tests/kserve_setup_external_access.sh kubeflow-user-example-com secure-model-predictor
159+
# Test external access pattern
160+
TOKEN=$(kubectl -n kubeflow-user-example-com create token default-editor)
161+
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
162+
-H "Authorization: Bearer $TOKEN" \
163+
-H "Content-Type: application/json" \
164+
"http://localhost:8080/kserve/kubeflow-user-example-com/secure-model-predictor/" \
165+
2>/dev/null || echo "404")
166+
if [ "$RESPONSE" != "404" ] && [ "$RESPONSE" != "200" ] && [ "$RESPONSE" != "503" ]; then
167+
exit 1
168+
fi
169+
170+
- name: Apply Pod Security Standards restricted levels
171+
run: ./tests/PSS_enable.sh
172+
173+
test-secure-authentication:
174+
runs-on: ubuntu-latest
175+
steps:
176+
- name: Checkout
177+
uses: actions/checkout@v4
178+
179+
- name: Install KinD, Create KinD cluster and Install kustomize
180+
run: ./tests/install_KinD_create_KinD_cluster_install_kustomize.sh
181+
182+
- name: Install kubectl
183+
run: ./tests/kubectl_install.sh
184+
185+
- name: Create kubeflow namespace
186+
run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
187+
188+
- name: Install Istio CNI
189+
run: ./tests/istio-cni_install.sh
190+
191+
- name: Install oauth2-proxy
192+
run: ./tests/oauth2-proxy_install.sh
193+
194+
- name: Install cert-manager
195+
run: ./tests/cert_manager_install.sh
196+
197+
- name: Install knative CNI (with secure cluster-local-gateway)
198+
run: ./tests/knative_install.sh
199+
200+
- name: Install KServe
201+
run: ./tests/kserve_install.sh
202+
203+
- name: Install KF Multi Tenancy
204+
run: ./tests/multi_tenancy_install.sh
205+
206+
- name: Install kubeflow-istio-resources
207+
run: kustomize build common/istio/kubeflow-istio-resources/base | kubectl apply -f -
208+
209+
- name: Create KF Profile
210+
run: ./tests/kubeflow_profile_install.sh
211+
212+
- name: Setup python 3.12
213+
uses: actions/setup-python@v4
214+
with:
215+
python-version: 3.12
216+
217+
- name: Port forward
218+
run: ./tests/port_forward_gateway.sh
219+
220+
- name: Verify JWT authentication policies are applied
221+
run: |
222+
kubectl get authorizationpolicy cluster-local-gateway-require-jwt -n istio-system
223+
kubectl get requestauthentication cluster-local-gateway-jwt -n istio-system
224+
kubectl get authorizationpolicy cluster-local-gateway -n istio-system
225+
kubectl get deployment cluster-local-gateway -n istio-system
226+
kubectl wait --for=condition=Available deployment/cluster-local-gateway -n istio-system --timeout=120s
227+
kubectl get pods -n istio-system -l app=cluster-jwks-proxy | grep -q Running || kubectl get pods -n istio-system -l app=cluster-jwks-proxy
228+
229+
- name: Wait for configurations to propagate
230+
run: sleep 60
231+
232+
- name: Run KServe secure authentication tests
233+
run: ./tests/kserve_jwt_authentication_test.sh kubeflow-user-example-com
234+
235+
- name: Apply Pod Security Standards restricted levels
236+
run: ./tests/PSS_enable.sh

‎applications/kserve/README.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,18 @@
22

33
For KServe installation and usage, see the [GitHub Actions tests](.github/workflows/kserve_test.yaml) which demonstrate working configurations.
44

5-
For complete documentation, visit the [official KServe website](https://kserve.github.io/website/).
5+
For complete documentation, visit the [official KServe website](https://kserve.github.io/website/).
6+
7+
## Integration with KubeFlow
8+
9+
When using KServe with path-based routing in a KubeFlow deployment, you may encounter VirtualService conflicts that result in 404 errors when accessing KServe InferenceServices.
10+
11+
**Common Issues:**
12+
- KServe InferenceServices return 404 errors when accessed via their configured domain
13+
- Conflicts between KubeFlow's wildcard VirtualServices and KServe's specific-host VirtualServices
14+
15+
**Solution:** See the [Istio troubleshooting guide](../../common/istio/README.md#virtualservice-conflicts-with-kserve-path-based-routing) for detailed resolution steps.
16+
17+
**Related Documentation:**
18+
- [KServe Path-Based Routing Configuration](https://kserve.github.io/website/docs/admin-guide/configurations#path-template)
19+
- [Upstream Istio Issue](https://github.com/istio/istio/issues/57404)

0 commit comments

Comments
 (0)