Enhance proxy handling and add IP name caching (#44)

* Enhance proxy handling and add IP name caching

- Added support for proxy name usage in connection requests to work around strict proxies.
- Enhanced test coverage for proxy functionality and integration tests.

* Remove output comments from example CA tests

Author: poupas

README.md

@@ -8,8 +8,8 @@ tunnel, allowing Probely to scan your internal applications.
 The Agent is open-source, and the code is freely available on the official
 [repository](https://github.com/probely/farcaster-onprem-agent).
 
-The following diagram shows an example network topology for Farcaster agent based 
-connectivity from a private client network (on-premise, private cloud, CI/CD, etc.) to 
+The following diagram shows an example network topology for Farcaster agent based
+connectivity from a private client network (on-premise, private cloud, CI/CD, etc.) to
 the Probely Cloud infrastructure.
 
 ![Farcaster high-level network architecture](./assets/img_Farcaster_Network_Overview.png)
@@ -44,18 +44,18 @@ means: *all ports from 1024 to 2048, inclusive*.
 Notes:
 
 1. `<agent-ip>` is the internal IP of the machine on your network where Probely's Farcaster Agent is running. The agent uses it to communicate with the Probely server.
-2. `<target-ip>` is the internal IP of your web application. 
+2. `<target-ip>` is the internal IP of your web application.
 If your target is configured to use internal extra-hosts, include their IPs here.
 The same goes for the target login URL if a different internal web application serves it.
 3. `<target-port>` is the service port of the server of your web application.
 Typical values are 80 and 443.
-4. The IP addresses of these hosts are subject to change. We recommend allowing 
+4. The IP addresses of these hosts are subject to change. We recommend allowing
 web access for the agent VM (HTTP and HTTPS ports). If this is not possible, the agent
 will use an HTTP proxy if you set the `HTTP_PROXY` variable.
 5. At this time, the hosts are: `registry.docker.io` and `registry-1.docker.io`
 6. This server receives connections from potentially vulnerable systems on your infrastructure.
 It is used, for example, to detect "Log4Shell"-type vulnerabilities.
-   
+
 # Installation
 
 The agent runs on a Docker container. It should work on any system with a Docker installation.
@@ -117,32 +117,32 @@ Probely's support team.
   Connecting to Probely (via UDP) ... done
   Setting local gateway rules     ... done
   Starting WireGuard gateway      ... done
-  
+
   Running...
   ```
 
-  Once up and running, the Agent in the Docker container knows the URL or IP of the target to scan from the target configuration in Probely. The Agent communicates with Probely to get this information before starting a scan.  
+  Once up and running, the Agent in the Docker container knows the URL or IP of the target to scan from the target configuration in Probely. The Agent communicates with Probely to get this information before starting a scan.
   Learn more about [how to scan internal applications with a Scanning Agent](https://help.probely.com/en/articles/4615595-how-to-scan-internal-applications-with-a-scanning-agent).
 
 ### Connection issues
   If the Agent is not connecting to Probely, please ensure that your [firewall](#network-requirements) is properly configured.
-  
-  Alternatively, the agent can use an HTTP proxy to connect to Probely if the `HTTP_PROXY` environment variable is set on the `docker-compose.yml` file.  
+
+  Alternatively, the agent can use a proxy to connect to Probely using standard environment variables. The agent honors `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` for outbound connections. HTTPS proxies are supported. `ALL_PROXY` is honored for WebSocket connections (ws://, wss://) via the standard library HTTP transport, but not for raw TCP connections.
   While the agent can use an HTTP proxy or a direct TCP connection to Probely, this can cause poor network performance. For more information, see this article about the [TCP Meltdown](https://web.archive.org/web/20220103191127/http://sites.inka.de/bigred/devel/tcp-tcp.html) problem. We **strongly recommend** that you allow the agent to connect to `54.247.135.113`,  `44.212.186.140`, and `54.253.10.194` on `UDP` port `443`.
 
 ### Unsuccessful UDP connection issues
-   If the Agent is not connecting through UDP, and you are getting the log: 
-   
+   If the Agent is not connecting through UDP, and you are getting the log:
+
    ```
    ...
    Connecting to Probely (via UDP)	... unsuccessful
    Configuring fallback TCP tunnel	... done
    Connecting to Probely (via TCP)	... done
    ...
    ```
-   
-   It's because the UDP connection is being blocked. 
-   
+
+   It's because the UDP connection is being blocked.
+
    To confirm if nothing is blocking the UDP connections, you can set up a UDP server using the following script **outside your network** to "echo" the received messages:
 
 ```python
@@ -163,8 +163,8 @@ if __name__ == "__main__":
     udp_server()
 ```
 
-And test it with: 
-    
+And test it with:
+
 ```shell
 $ echo "AAAAAA" | nc -w 3 -u xx.xx.xx.xx 12345
 Received your message: AAAAAA

compose/docker-compose.yml

@@ -15,7 +15,10 @@ services:
     #   Probely's API URL. If not set, the Agent will use the API URLs for all regions.
     #
     #   FARCASTER_FORCE_TCP
-    #   If set to true, the Agent will use TCP to connect to Probely.
+    #   If set to true, the Agent will use TCP to connect to Probely/Snyk.
+    #
+    #   FARCASTER_PROXY_NAMES
+    #   If set to true, the Agent will use hostnames in proxy requests when available.
     #
     #   HTTP_PROXY (optional)
     #   An advanced option that can be used to configure an HTTP proxy for the Agent to connect to Probely.
@@ -25,10 +28,11 @@ services:
     - FARCASTER_API_URL
     - HTTP_PROXY
     - FARCASTER_FORCE_TCP
+    - FARCASTER_PROXY_NAMES
     tmpfs:
     - /run
     cap_add:
-      # Required for kernel support. If you remove this, the Agent will fall back to a userspace TCP/IP stack.
+      # Required to use the kernel WireGuard implementation. If you remove this, the Agent will fall back to a userspace implementation.
       - NET_ADMIN
     restart: unless-stopped
 

farcaster-go/agent/agent.go

@@ -97,26 +97,29 @@ type Agent struct {
 	conns  atomic.Uint32
 	cancel chan struct{}
 	log    *zap.SugaredLogger
-	// WaitGroup to track background goroutines
+	// WaitGroup to track background goroutines.
 	wg sync.WaitGroup
-	// Flag to track if Close has already been called
+	// Flag to track if Close has already been called.
 	closing atomic.Bool
-	// Enable IPv6 DNS resolution
+	// Enable IPv6 DNS resolution.
 	useIPv6 bool
+	// Use hostnames in proxy requests when available.
+	proxyUseNames bool
 }
 
 // New creates a new agent.
-func New(token string, apiURLs []string, logger *zap.SugaredLogger, useIPv6 bool) *Agent {
+func New(token string, apiURLs []string, logger *zap.SugaredLogger, useIPv6 bool, proxyUseNames bool) *Agent {
 	if logger == nil {
 		logger = zap.NewNop().Sugar()
 	}
 	return &Agent{
-		State:   newState(),
-		token:   token,
-		apiURLs: apiURLs,
-		cancel:  make(chan struct{}, 1), // Use buffered channel to ensure signal is not lost
-		log:     logger,
-		useIPv6: useIPv6,
+		State:         newState(),
+		token:         token,
+		apiURLs:       apiURLs,
+		cancel:        make(chan struct{}, 1), // Use buffered channel to ensure signal is not lost
+		log:           logger,
+		useIPv6:       useIPv6,
+		proxyUseNames: proxyUseNames,
 	}
 }
 
@@ -234,7 +237,7 @@ func (a *Agent) up(bind conn.Bind) error {
 	// route traffic from remote peers to the local network without requiring
 	// special privileges or devices (e.g. /dev/net/tun).
 	mtu := min(gwCfg.MTU, 1340)
-	a.gw, err = netstack.NewTUN(addr, "gateway", mtu, a.log, a.useIPv6)
+	a.gw, err = netstack.NewTUN(addr, "gateway", mtu, a.log, a.useIPv6, a.proxyUseNames)
 	if err != nil {
 		return fmt.Errorf("failed to create gateway: %w", err)
 	}

farcaster-go/agent/agent_test.go

@@ -10,9 +10,13 @@ import (
 var token = os.Getenv("FARCASTER_AGENT_TOKEN")
 
 func TestAgentLifecycle(t *testing.T) {
+	if token == "" {
+		t.Skip("Skipping TestAgentLifecycle: FARCASTER_AGENT_TOKEN not set")
+	}
 	logger := zap.NewNop().Sugar()
 	useIPv6 := false
-	a := New(token, nil, logger, useIPv6)
+	proxyUseNames := false
+	a := New(token, nil, logger, useIPv6, proxyUseNames)
 	if a.CheckToken() != nil {
 		t.Error("Valid token considered invalid")
 	}
@@ -38,7 +42,7 @@ func TestAgentLifecycle(t *testing.T) {
 	t.Logf("Closing agent a...")
 	a.Close()
 
-	b := New(token, nil, logger, useIPv6)
+	b := New(token, nil, logger, useIPv6, proxyUseNames)
 	if b.CheckToken() != nil {
 		t.Error("Valid token considered invalid")
 	}

farcaster-go/cmd/farcasterd/root.go

@@ -6,6 +6,7 @@ import (
 	"os/signal"
 	"path/filepath"
 	"runtime"
+	"strconv"
 	"strings"
 	"syscall"
 	"time"
@@ -29,16 +30,17 @@ const (
 )
 
 type agentConfig struct {
-	token      string
-	apiURLs    []string
-	checkToken bool
-	controlAPI string
-	group      string
-	showVers   bool
-	logPath    string
-	debug      bool
-	apiURL     string
-	ipv6       bool
+	token         string
+	apiURLs       []string
+	checkToken    bool
+	controlAPI    string
+	group         string
+	showVers      bool
+	logPath       string
+	debug         bool
+	apiURL        string
+	ipv6          bool
+	proxyUseNames bool
 }
 
 var (
@@ -94,7 +96,16 @@ func init() {
 	rootCmd.PersistentFlags().BoolVarP(&appCfg.showVers, "version", "v", false, "Print the version and exit")
 	rootCmd.PersistentFlags().StringVarP(&appCfg.logPath, "log", "l", "", "Log file path. Log to stderr if not specified")
 	rootCmd.PersistentFlags().BoolVarP(&appCfg.debug, "debug", "d", false, "Enable debug logging")
+	// Default from env var if present
+	defaultProxyUseNames := false
+	if v := strings.TrimSpace(os.Getenv("FARCASTER_PROXY_NAMES")); v != "" {
+		if b, err := strconv.ParseBool(v); err == nil {
+			defaultProxyUseNames = b
+		}
+	}
+
 	rootCmd.PersistentFlags().BoolVarP(&appCfg.ipv6, "ipv6", "", false, "Enable IPv6/AAAA DNS query resolution")
+	rootCmd.PersistentFlags().BoolVar(&appCfg.proxyUseNames, "proxy-names", defaultProxyUseNames, "Use hostnames instead of IPs in proxy CONNECT/SOCKS5 requests")
 }
 
 // Execute runs the agent.
@@ -126,7 +137,7 @@ func runAgent(cfg agentConfig) {
 
 	// If the --check-token flag is set, we just check if the token is valid.
 	if cfg.checkToken {
-		a := agent.New(cfg.token, cfg.apiURLs, logger, cfg.ipv6)
+		a := agent.New(cfg.token, cfg.apiURLs, logger, cfg.ipv6, cfg.proxyUseNames)
 		if err := a.CheckToken(); err != nil {
 			logger.Errorf("Token validation failed: %v", err)
 			exit(1)
@@ -151,7 +162,7 @@ func runAgent(cfg agentConfig) {
 	}
 
 	startAgent := func() error {
-		a := agent.New(cfg.token, cfg.apiURLs, logger, cfg.ipv6)
+		a := agent.New(cfg.token, cfg.apiURLs, logger, cfg.ipv6, cfg.proxyUseNames)
 		if err := a.ConnectWait(maxConnTries); err != nil {
 			a.Close()
 			return err

farcaster-go/config/config.go

@@ -25,44 +25,17 @@ import (
 	"golang.org/x/crypto/nacl/secretbox"
 	"probely.com/farcaster/netutils"
 	"probely.com/farcaster/settings"
+	"probely.com/farcaster/tlsconfig"
 
 	"github.com/mr-tron/base58"
 )
 
 const (
-	defaultMTU  = 1420
-	defaultPort = 0
-	timeout     = time.Second * 30
+	defaultMTU     = 1420
+	defaultPort    = 0
+	defaultTimeout = time.Second * 30
 )
 
-var (
-	configClient = createHTTPClient()
-)
-
-// createHTTPClient creates an HTTP client with the appropriate TLS configuration
-func createHTTPClient() *http.Client {
-	// Check if certificate verification should be skipped
-	skipVerify := false
-	if val := os.Getenv("FARCASTER_SKIP_CERT_VERIFY"); val != "" {
-		switch strings.ToLower(val) {
-		case "1", "ok", "true", "yes", "enable", "enabled":
-			skipVerify = true
-		}
-	}
-
-	transport := &http.Transport{
-		TLSClientConfig: &tls.Config{
-			InsecureSkipVerify: skipVerify,
-		},
-		Proxy: http.ProxyFromEnvironment,
-	}
-
-	return &http.Client{
-		Timeout:   timeout,
-		Transport: transport,
-	}
-}
-
 // WireGuardConfig represents a WireGuard configuration. Not all fields are
 // supported.
 type WireGuardConfig struct {
@@ -327,6 +300,21 @@ func (c *FarcasterConfig) resolveEndpoint(peer *Peer) error {
 	return fmt.Errorf("could not resolve host %s. All resolution attempts failed", host)
 }
 
+func (c *FarcasterConfig) getHTTPClient(timeout time.Duration) *http.Client {
+	tlsConfig, err := tlsconfig.GetTLSConfig()
+	if err != nil {
+		c.log.Warnf("Error getting TLS config: %v", err)
+		tlsConfig = &tls.Config{}
+	}
+	return &http.Client{
+		Timeout: timeout,
+		Transport: &http.Transport{
+			TLSClientConfig: tlsConfig,
+			Proxy:           http.ProxyFromEnvironment,
+		},
+	}
+}
+
 // Fetch the configuration from the API.
 func (c *FarcasterConfig) fetch(url string) ([]byte, error) {
 	var data []byte
@@ -352,7 +340,8 @@ func (c *FarcasterConfig) fetch(url string) ([]byte, error) {
 	req.Header.Set("User-Agent", userAgent)
 
 	// Send the request.
-	resp, err := configClient.Do(req)
+	client := c.getHTTPClient(defaultTimeout)
+	resp, err := client.Do(req)
 	if err != nil {
 		return nil, err
 	}