Fix timeout flag inconsistency (#1560)

[1234-ad]

Description

This PR fixes the timeout flag inconsistency issue reported in #1560 where the -timeout option does not work consistently across runs.

Root Cause

The issue was caused by:

1. Use of deprecated Dial method instead of DialContext
2. Missing timeout configurations for TLS handshake and response headers
3. Lack of KeepAlive timeout which could cause connections to hang

Changes Made

• ✅ Replaced deprecated Dial with DialContext for proper context-aware timeout handling
• ✅ Added TLSHandshakeTimeout to ensure TLS handshakes respect the timeout
• ✅ Added ResponseHeaderTimeout to prevent hanging on slow response headers
• ✅ Added KeepAlive timeout to the dialer to prevent connection hanging
• ✅ Added ExpectContinueTimeout for better HTTP/1.1 handling
• ✅ Consolidated timeout duration calculation to ensure consistency

Testing

The fix ensures that all HTTP operations (connection, TLS handshake, response headers) respect the configured timeout value consistently across all runs.

Related Issues

Fixes #1560
Related to #872

Checklist

• Code follows the project's coding standards
• Changes are backward compatible
• Commit message follows conventional commits format
• No breaking changes introduced

Summary by CodeRabbit

• Chores
  • Improved HTTP timeout handling with more granular controls for connection establishment, TLS handshakes, and response headers to enhance reliability and stability of network requests.

[coderabbitai]

Walkthrough

The change modifies HTTP transport and client timeout configurations in the subscraping agent by replacing Dial with DialContext and introducing granular timeout parameters (KeepAlive, TLSHandshakeTimeout, ResponseHeaderTimeout, ExpectContinueTimeout) to improve timeout consistency across HTTP operations.

Changes

Cohort / File(s)	Summary
HTTP Timeout Configuration pkg/subscraping/agent.go	Replaces Dial with DialContext in http.Transport, adds granular timeout settings (TLSHandshakeTimeout, ResponseHeaderTimeout, ExpectContinueTimeout, KeepAlive), and updates http.Client Timeout to use timeoutDuration for more consistent timeout enforcement.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A rabbit once waited so long,
But timeouts would come out all wrong,
Now DialContext flows,
With timeouts that prose,
Each second now sings the right song! 🐰⏱️

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Fix timeout flag inconsistency (#1560)' clearly references the primary issue and accurately summarizes the main change in the pull request.
Linked Issues check	✅ Passed	The code changes directly address all requirements from #1560: replacing deprecated Dial with DialContext, adding TLSHandshakeTimeout, ResponseHeaderTimeout, KeepAlive, and ExpectContinueTimeout to ensure consistent timeout handling across all network operations.
Out of Scope Changes check	✅ Passed	All changes in pkg/subscraping/agent.go are focused on timeout handling implementation and are directly related to fixing the timeout flag inconsistency issue; no out-of-scope changes detected.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[dogancanbakir]

Closing. reason: #1723