fix(raw): handle full URLs in unsafe raw requests

[dwisiswant0]

Proposed changes

fix(raw): handle full URLs in unsafe raw requests

Previously, when using unsafe: true with full
URLs (e.g., GET http://example.com/path HTTP/1.1),
the Parse func would treat the full URL as a
relative path, resulting in malformed requests
like GET /http://example.com/path HTTP/1.1.

This occurred because the full URL handling
logic was only executed for non-unsafe requests,
causing unsafe requests with full URLs to fall
through to the unsafe case which wasn't designed
to handle them.

Changes:

• Extract full URL handling before mode-specific
  logic runs.
• Convert full URLs to relative paths for both
  safe and unsafe modes.
• Update UnsafeRawBytes with the correct
  relative path when unsafe is true.
• Ensure path merging works correctly with
  disable-path-automerge.

This fix maintains backward compatibility while
properly supporting the previously broken
combination of unsafe mode with full URLs.

Fixes #6558.

Proof

• unsafe-raw-request.yaml:

  id: unsafe-raw-request
  
  info:
    name: unsafe-raw-request
    author: dwisiswant0
    severity: info
    tags: test
  
  http:
    - raw:
        - |+
          GET http://127.0.0.1/foo{{Path}} HTTP/1.1
          Host: {{Hostname}}
          User-Agent: Mozilla/5.0
          Connection: close
  
      disable-path-automerge: true
      unsafe: true

• unsafe-raw-request-merge.yaml:

  id: unsafe-raw-request-merge
  
  info:
    name: unsafe-raw-request with path merge
    author: dwisiswant0
    severity: info
    tags: test
  
  http:
    - raw:
        - |+
          GET http://127.0.0.1/api{{Path}} HTTP/1.1
          Host: {{Hostname}}
          User-Agent: Mozilla/5.0
          Connection: close
  
      disable-path-automerge: false
      unsafe: true

• safe-with-full-url-merge.yaml:

  id: safe-with-full-url-merge
  
  info:
    name: safe mode with full URL and merge
    author: dwisiswant0
    severity: info
    tags: test
  
  http:
    - raw:
        - |+
          GET http://example.com/api/users{{Path}} HTTP/1.1
          Host: {{Hostname}}
          User-Agent: Mozilla/5.0
          Connection: close
  
      disable-path-automerge: false
      unsafe: false

Outputs:

$ ./bin/nuclei -u "http://httpbin.org/get" -t unsafe-raw-request.yaml -debug-req 2>&1 | grep -A4 GET
GET /foo/ HTTP/1.1
Host: httpbin.org
User-Agent: Mozilla/5.0
Connection: close

$ ./bin/nuclei -u "http://httpbin.org/get" -t unsafe-raw-request-merge.yaml -debug-req 2>&1 | grep -A4 GET
GET /get/api/ HTTP/1.1
Host: httpbin.org
User-Agent: Mozilla/5.0
Connection: close

$ ./bin/nuclei -u "http://httpbin.org/get" -t safe-with-full-url-merge.yaml -debug-req 2>&1 | grep -A5 GET
GET /get/api/users/ HTTP/1.1
Host: httpbin.org
User-Agent: Mozilla/5.0
Connection: close
Accept-Encoding: gzip

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• Tests

  • Added comprehensive test coverage for HTTP request handling with full URLs, including safe/unsafe modes, HTTPS, path merging, and query parameter scenarios.

• Refactor

  • Optimized internal logic for processing full URLs in HTTP requests with consolidated handling.

[coderabbitai]

Walkthrough

The Parse function in the HTTP raw protocol handler is refactored to handle full URLs upfront: when a path begins with http:// or https://, it is immediately parsed, converted to a relative path, optionally replaced in unsafe bytes, and set to the relative form before proceeding. A duplicate switch-case branch is removed.

Changes

Cohort / File(s)	Summary
URL handling consolidation pkg/protocols/http/raw/raw.go	Moved full URL detection and parsing to the beginning of Parse function; consolidated logic for extracting relative paths from full URLs; removed redundant switch case branch; unsafe path replacement now applied earlier in the flow.
Test coverage for URL handling pkg/protocols/http/raw/raw_test.go	Added comprehensive test cases covering unsafe/safe mode handling of full URLs, relative path extraction, host-target path merging with query parameters, HTTPS URLs, and root path edge cases with path automerge disabled.

Sequence Diagram(s)

sequenceDiagram
    participant Parse as Parse Function
    participant URL as URL Parser
    participant Path as Path Handler

    Parse->>Parse: Check if Path starts with http:// or https://
    alt Full URL detected
        Parse->>URL: Parse full URL
        URL->>Parse: Return parsed components
        Parse->>Path: Extract relative path
        alt Unsafe mode
            Parse->>Parse: Replace path in UnsafeRawBytes
        end
        Parse->>Parse: Set Path to relative path
    end
    Parse->>Parse: Continue with main switch logic (non-URL branches)

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

• Logic consolidation is straightforward: the refactor moves URL handling earlier without changing core behavior
• Tests are well-structured: comprehensive coverage validates both unsafe and safe modes, edge cases with query parameters and root paths
• Low risk of regression: previous switch-case branch is simply relocated, minimizing logic changes
• Areas to verify:
  • Confirm UnsafeRawBytes mutation only occurs when unsafe flag is true
  • Verify relative path extraction correctly handles URLs with query parameters and fragments
  • Check that root path handling (/ vs empty string) behaves consistently in all modes

Poem

🐰 A rabbit hops through URLs with glee,
No slashes doubled where they shouldn't be!
Full paths are parsed before the switch,
Each test ensures there's nary a glitch. ✨

Pre-merge checks and finishing touches

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main fix: handling full URLs in unsafe raw requests, which is exactly what the code changes address.
Linked Issues check	✅ Passed	The PR directly addresses issue #6558 by moving full-URL handling before mode-specific logic, ensuring both safe and unsafe modes correctly convert full URLs to relative paths while respecting disable-path-automerge.
Out of Scope Changes check	✅ Passed	All changes are scoped to raw HTTP request parsing: consolidating full-URL handling before the switch statement and adding comprehensive test coverage for unsafe and safe mode behaviors with full URLs.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dwisiswant0/fix/raw/handle-full-URLs-in-unsafe-raw-requests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

pkg/protocols/http/raw/raw.go (1)

41-58: Excellent refactoring that consolidates full URL handling.

This upfront preprocessing correctly extracts relative paths from full URLs before the switch logic, fixing issue #6558. The approach ensures both safe and unsafe modes handle full URLs consistently.

The use of bytes.Replace(..., 1) on line 53 is appropriate since the path appears first in the request line (e.g., "GET http://example.com/path HTTP/1.1"), so only the first occurrence is replaced. This preserves any other occurrences that might appear in headers or body.

Consider adding a brief comment on line 53 explaining why count=1 is used:

 		if unsafe {
+			// Replace only the first occurrence (in the request line), preserving any other occurrences in headers/body
 			rawrequest.UnsafeRawBytes = bytes.Replace(rawrequest.UnsafeRawBytes, []byte(prevPath), []byte(relPath), 1)
 		}

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d7da0e0 and d5e65d1.

📒 Files selected for processing (2)

• pkg/protocols/http/raw/raw.go (1 hunks)
• pkg/protocols/http/raw/raw_test.go (1 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt
Run static analysis with go vet

Files:

• pkg/protocols/http/raw/raw.go
• pkg/protocols/http/raw/raw_test.go

pkg/protocols/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()

Files:

• pkg/protocols/http/raw/raw.go
• pkg/protocols/http/raw/raw_test.go

🧠 Learnings (1)

📚 Learning: 2025-06-30T16:33:26.746Z

Learnt from: dwisiswant0
Repo: projectdiscovery/nuclei PR: 6290
File: pkg/fuzz/component/path_test.go:43-44
Timestamp: 2025-06-30T16:33:26.746Z
Learning: The user dwisiswant0 declined adding documentation for internal API changes in pkg/fuzz/component/path when the change was from nested URL field access (rebuilt.URL.Path, rebuilt.URL.String()) to direct field access (rebuilt.Path, rebuilt.String()), indicating they don't consider additional documentation necessary for such internal API modifications.

Applied to files:

• pkg/protocols/http/raw/raw.go

🧬 Code graph analysis (1)

pkg/protocols/http/raw/raw_test.go (1)

pkg/protocols/http/raw/raw.go (1)

• Parse (35-142)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Lint

🔇 Additional comments (6)

pkg/protocols/http/raw/raw_test.go (6)

125-135: LGTM! Core fix validated.

This test correctly verifies that full URLs in unsafe raw requests are converted to relative paths, addressing the bug described in issue #6558.

---

137-147: LGTM! Path merging behavior validated.

This test correctly verifies that when disable-path-automerge is false, the extracted relative path is merged with the target URL's path.

---

149-159: LGTM! Query parameter preservation validated.

This test correctly verifies that query parameters are preserved when converting full URLs to relative paths.

---

161-171: LGTM! HTTPS URL handling validated.

This test confirms that HTTPS full URLs are handled identically to HTTP, which is the correct behavior.

---

173-189: LGTM! Root path edge case validated.

This test correctly handles the root path edge case for both automerge modes, confirming the existing behavior is preserved.

---

191-199: LGTM! Safe mode backward compatibility validated.

This test confirms that safe mode continues to work correctly with full URLs, maintaining backward compatibility.

[Mzack9999]

lgtm! Looks like the only way to send a full URL even as unsafe it's set self-contained to true