How hard would it be to extend dumbproxy to cache/dump files?

[voltagex]

Hi,
I'm still going through the code trying to see how it all works, but how hard would it be to extend dumbproxy to cache files? My use case would be modifying dumbproxy to be a transparent proxy for a few VMs and containers to save me download time on Debian packages, for example.

[Snawoot]

how hard would it be to extend dumbproxy to cache files?

This function is responsible for handling of plaintext non-tunneled requests:

dumbproxy/handler/handler.go

Lines 140 to 171 in de1cd59

	func (s *ProxyHandler) HandleRequest(wr http.ResponseWriter, req *http.Request, username string) {
	req.RequestURI = ""
	forwardReqBody := newH1ReqBodyPipe()
	origBody := req.Body
	req.Body = forwardReqBody.Body()
	go func() {
	s.forward(req.Context(), username, wrapH1ReqBody(origBody), forwardReqBody)
	}()
	if req.ProtoMajor == 2 {
	req.URL.Scheme = "http" // We can't access :scheme pseudo-header, so assume http
	req.URL.Host = req.Host
	}
	resp, err := s.httptransport.RoundTrip(req)
	if err != nil {
	var accessErr derrors.ErrAccessDenied
	if errors.As(err, &accessErr) {
	s.logger.Warning("Access denied: %v", err)
	http.Error(wr, "Access denied", http.StatusForbidden)
	return
	}
	s.logger.Error("HTTP fetch error: %v", err)
	http.Error(wr, "Server Error", http.StatusInternalServerError)
	return
	}
	defer resp.Body.Close()
	s.logger.Info("%v %v %v %v", req.RemoteAddr, req.Method, req.URL, resp.Status)
	delHopHeaders(resp.Header)
	copyHeader(wr.Header(), resp.Header)
	wr.WriteHeader(resp.StatusCode)
	flush(wr)
	s.forward(req.Context(), username, wrapH1RespWriter(wr), wrapH1ReqBody(resp.Body))
	}

Probably you can just add cache there. I recommend extend the transport and wrap http.Transport used in there with something implementing http.RoundTripper interface and doing proper caching.

My use case would be modifying dumbproxy to be a transparent proxy

Transparent part probably would require changes here and there or some adapter in front of dumbproxy to intercept HTTP connections and forward them to a HTTP proxy without breaking HTTP semantics.

for a few VMs and containers to save me download time on Debian packages, for example.

Not sure if this will make sense nowadays or in the nearest future. A lot of repos have now HTTPS transport and it won't be as simple to intercept it.

On the other hand, there are already specialized off-the-shelf solutions for that problem like AptCacherNg (here is a fairly recent blog post about it).

Another direction to look at is to use just Squid for that purpose, it was developed as a webcache from day one. On the contrary, dumbproxy is made for post-HTTP world.

Finally, it caching can be achieved with some caching reverse proxy like nginx: https://www.apalrd.net/posts/2024/cluster_debcache/
I think it will be not too hard to figure out transparent rewriting on top of that.

[voltagex]

I really appreciate this answer. I originally started looking at this after being frustrated by squid.

Thanks!