get forward header for proxies and apply it in Jetty #11386

DaanHoogland · 2025-08-04T08:14:20Z

Description

This PR...

Fixes: #11387

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Copilot

Pull Request Overview

This PR addresses proxy header handling by making proxy configuration accessible to Jetty and configuring forwarded request handling. The changes focus on fixing issue #11375 related to proxy headers not being properly applied in the Jetty server configuration.

Changed visibility of proxy header configuration keys from package-private to public
Added Jetty ForwardedRequestCustomizer configuration to handle proxy headers
Implemented automatic detection and application of the first configured forward header

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
ApiServer.java	Made proxy header configuration keys public to allow access from other packages
ServerDaemon.java	Added Jetty ForwardedRequestCustomizer with automatic header configuration from ApiServer settings

client/src/main/java/org/apache/cloudstack/ServerDaemon.java

DaanHoogland · 2025-08-04T08:27:58Z

@blueorangutan package

blueorangutan · 2025-08-04T08:30:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sureshanaparti

clgtm

codecov · 2025-08-04T09:00:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 17.35%. Comparing base (d4229d3) to head (5e56dfd).
⚠️ Report is 29 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##              main   #11386       +/-   ##
============================================
+ Coverage     3.63%   17.35%   +13.71%     
- Complexity       0    15230    +15230     
============================================
  Files          441     5885     +5444     
  Lines        36961   525625   +488664     
  Branches      6771    64160    +57389     
============================================
+ Hits          1344    91218    +89874     
- Misses       35457   424111   +388654     
- Partials       160    10296    +10136

Flag	Coverage Δ
uitests	`3.63% <ø> (-0.01%)`	⬇️
unittests	`18.39% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2025-08-04T10:31:43Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14532

rosi-shapeblue

LGTM - PR tested and verified

Test Results - developed a custom script to verify the fix:
test_forwarded_headers.sh.txt

Specific Test Cases Passed

Single forwarded header: 203.0.113.1 → Logged correctly
Multiple headers: X-Forwarded-For prioritized over HTTP_CLIENT_IP
Comma-separated IPs: 203.0.113.20, 192.168.1.100 → First IP (203.0.113.20) logged
Control test: No headers → Localhost IP logged (expected behavior)

Test Environment WITHOUT PR #11386 (Control)

[root@ref-trl-9173-k-Mol8-rositsa-kyuchukova-mgmt1 ~]# ./test_forwarded_headers.sh 
=== CloudStack Forwarded Headers Test (PR #11386) ===
Testing on: ref-trl-9173-k-Mol8-rositsa-kyuchukova-mgmt1
Date: Thu Aug  7 08:33:59 UTC 2025

Clearing CloudStack access log...
Waiting 2 seconds for log clearing...
Running test API calls with forwarded headers...

Test 1: Sending X-Forwarded-For: 203.0.113.1
Test 2: Sending HTTP_CLIENT_IP: 203.0.113.2
Test 3: Sending HTTP_X_FORWARDED_FOR: 203.0.113.3
Test 4: Sending multiple headers (X-Forwarded-For: 203.0.113.10, HTTP_CLIENT_IP: 203.0.113.11)
Test 5: Control test - no forwarded headers
Test 6: Comma-separated X-Forwarded-For: 203.0.113.20, 192.168.1.100

Waiting 3 seconds for all requests to be logged...

=== RESULTS ANALYSIS ===
CloudStack access log entries:
==============================
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:01] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:02] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:03] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:04] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:05] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:06] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"

=== VALIDATION ===
==================
ISSUE: No forwarded IP addresses (203.0.113.x) found in access log
This suggests PR #11386 changes are not working correctly

Troubleshooting info:
====================
All log entries:
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:01] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:02] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:03] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:04] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:05] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 08:34:06] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"

Test completed at: Thu Aug  7 08:34:09 UTC 2025
===========================

Results from acces.log

0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:51] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:52] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:53] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:54] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:55] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"
0:0:0:0:0:0:0:1 - - [2025-08-07 09:06:56] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1"

Environment WITH PR #11386 (Test)

[root@ref-trl-9184-k-Mol8-rositsa-kyuchukova-mgmt1 ~]# ./test_forwarded_headers.sh 
=== CloudStack Forwarded Headers Test (PR #11386) ===
Testing on: ref-trl-9184-k-Mol8-rositsa-kyuchukova-mgmt1
Date: Thu Aug  7 08:32:18 UTC 2025

Clearing CloudStack access log...
Waiting 2 seconds for log clearing...
Running test API calls with forwarded headers...

Test 1: Sending X-Forwarded-For: 203.0.113.1
Test 2: Sending HTTP_CLIENT_IP: 203.0.113.2
Test 3: Sending HTTP_X_FORWARDED_FOR: 203.0.113.3
Test 4: Sending multiple headers (X-Forwarded-For: 203.0.113.10, HTTP_CLIENT_IP: 203.0.113.11)
Test 5: Control test - no forwarded headers
Test 6: Comma-separated X-Forwarded-For: 203.0.113.20, 192.168.1.100

Waiting 3 seconds for all requests to be logged...

=== RESULTS ANALYSIS ===
CloudStack access log entries:
==============================
10.0.34.38 - - [07/Aug/2025:08:32:19 +0000] "GET /client/api?apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listHosts&response=json&signature=BYVU5RhMdLyIIO33ThMJS7BwqWk%3D HTTP/1.1" 200 1328 "-" "Go-http-client/1.1" 31
203.0.113.1 - - [07/Aug/2025:08:32:20 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:08:32:21 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:08:32:22 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 3
203.0.113.10 - - [07/Aug/2025:08:32:23 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:08:32:24 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
203.0.113.20 - - [07/Aug/2025:08:32:25 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1

=== VALIDATION ===
==================
SUCCESS: Forwarded IP addresses found in CloudStack access log!
PR #11386 forwarded headers functionality is working correctly

Detected forwarded IPs:
======================
  203.0.113.1 - - [07/Aug/2025:08:32:20 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
  203.0.113.10 - - [07/Aug/2025:08:32:23 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
  203.0.113.20 - - [07/Aug/2025:08:32:25 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1

Analysis:
=========
Unique forwarded IPs detected:
  203.0.113.1 (appeared 2 time(s))
  203.0.113.10 (appeared 1 time(s))
  203.0.113.20 (appeared 1 time(s))

Test completed at: Thu Aug  7 08:32:28 UTC 2025
===========================

Results from access.log

203.0.113.1 - - [07/Aug/2025:09:08:59 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 2
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:09:09:00 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:09:09:01 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
203.0.113.10 - - [07/Aug/2025:09:09:02 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
[0:0:0:0:0:0:0:1] - - [07/Aug/2025:09:09:03 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1
203.0.113.20 - - [07/Aug/2025:09:09:04 +0000] "GET /client/api?command=listCapabilities&response=json HTTP/1.1" 401 133 "-" "curl/7.61.1" 1

* get forward header and apply it fro proxies Co-authored-by: Daan Hoogland <[email protected]>

get forward header and apply it fro proxies

3362d5a

boring-cyborg bot added the component:management-server label Aug 4, 2025

DaanHoogland requested review from NuxRo, Copilot and nvazquez August 4, 2025 08:14

DaanHoogland added this to the 4.21.1 milestone Aug 4, 2025

Copilot AI reviewed Aug 4, 2025

View reviewed changes

spelloos

6bf4473

sureshanaparti approved these changes Aug 4, 2025

View reviewed changes

sureshanaparti added the status:needs-testing label Aug 4, 2025

rosi-shapeblue self-assigned this Aug 6, 2025

use StringUtils.getPreferredCharset()

a08b725

rosi-shapeblue approved these changes Aug 7, 2025

View reviewed changes

DaanHoogland removed the status:needs-testing label Aug 7, 2025

DaanHoogland marked this pull request as ready for review August 7, 2025 09:39

DaanHoogland mentioned this pull request Aug 7, 2025

[Improvement] Use client IP addresses instead of proxy servers IPs on the logs #11375

Closed

import

5e56dfd

sureshanaparti modified the milestones: 4.21.1, 4.21.0 Aug 7, 2025

sureshanaparti added this to Apache CloudStack 4.21.0 Aug 7, 2025

sureshanaparti moved this to In Progress in Apache CloudStack 4.21.0 Aug 7, 2025

sureshanaparti merged commit fe8f3c8 into apache:main Aug 7, 2025
26 checks passed

github-project-automation bot moved this from In Progress to Done in Apache CloudStack 4.21.0 Aug 7, 2025

sureshanaparti deleted the ghi11375-ipLogging branch August 7, 2025 12:47

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Sep 4, 2025

get forward header for proxies and apply it in Jetty (apache#11386)

2a8d113

* get forward header and apply it fro proxies Co-authored-by: Daan Hoogland <[email protected]>

abh1sar mentioned this pull request Oct 17, 2025

Fix logging of forwarded IPs in logs #11854

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

get forward header for proxies and apply it in Jetty #11386

get forward header for proxies and apply it in Jetty #11386

Uh oh!

DaanHoogland commented Aug 4, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DaanHoogland commented Aug 4, 2025

Uh oh!

blueorangutan commented Aug 4, 2025

Uh oh!

sureshanaparti left a comment

Uh oh!

codecov bot commented Aug 4, 2025 •

edited

Loading

Uh oh!

blueorangutan commented Aug 4, 2025

Uh oh!

rosi-shapeblue left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

get forward header for proxies and apply it in Jetty #11386

get forward header for proxies and apply it in Jetty #11386

Uh oh!

Conversation

DaanHoogland commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DaanHoogland commented Aug 4, 2025

Uh oh!

blueorangutan commented Aug 4, 2025

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Aug 4, 2025

Uh oh!

rosi-shapeblue left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DaanHoogland commented Aug 4, 2025 •

edited

Loading

codecov bot commented Aug 4, 2025 •

edited

Loading

rosi-shapeblue left a comment •

edited

Loading