[Internal Instance] - Enable support for Cloudflare's Privacy Gateway

[paolodamico]

As the indexer and gateway instances get rolled out, they'll have Cloudflare's Privacy Gateway (docs) in front for additional enhanced privacy. Neither service will ever learn the end user's IP address.

Naturally the user may still choose to use a different gateway/indexer service or none at all.

[paolodamico]

@alessandromazza98 @karankurbur maybe you can look into this?

[alessandromazza98]

@alessandromazza98 @karankurbur maybe you can look into this?

similar here, I can take it as well. I'll be flying to SF tomorrow so I don't expect to start it tomorrow

[alessandromazza98]

after a while exploring this I think we have two options:

Option A: Separate OHTTP Gateway (Sidecar/Reverse Proxy)

Pros:

• Zero code changes to our existing gateway/indexer services
• Separation of concerns - OHTTP handling is isolated
• Easy to enable/disable - just route traffic through or bypass the sidecar
• Battle-tested - Cloudflare's Go implementation is production-ready
• Simpler debugging - plain HTTP between sidecar and our services

Cons:

• Additional service to deploy/maintain - more infrastructure complexity
• Extra network hop - slightly higher latency (~1-5ms)
• Resource overhead - another container running
• TLS termination considerations - need to decide where TLS happens

flowchart LR
    Client -->|OHTTP| CloudflareRelay
    CloudflareRelay -->|OHTTP| OHTTPSidecar
    OHTTPSidecar -->|Plain HTTP| Gateway
    OHTTPSidecar -->|Plain HTTP| Indexer

Loading

Option B: Integrated OHTTP Middleware in Axum

Pros:

• Single deployment - no extra services
• Lower latency - no extra network hop
• Unified logging/metrics - all in one place
• Less infrastructure - simpler Docker Compose / Kubernetes setup

Cons:

• Code changes required - need to add middleware to both services
• Rust ohttp crate maturity - less battle-tested than Cloudflare's Go implementation
• Tighter coupling - OHTTP logic mixed with business logic
• Duplicated work - need to implement in both gateway and indexer
• Harder to disable - requires code changes to bypass

flowchart LR
    Client -->|OHTTP| CloudflareRelay
    CloudflareRelay -->|OHTTP| Gateway
    CloudflareRelay -->|OHTTP| Indexer
    
    subgraph services [Our Services]
        Gateway
        Indexer
    end

Loading

My take

Currently I'm leaning toward option A for three main reasons:

• the Cloudflare go impl is more battle tested and already production level.
• it doesn't require any modification to our services, which can be introduce further issues / bugs
• it's super easy to enable / disable it if we need to for any reason

cc @paolodamico do you have something else in mind or do you have other thoughts?

[paolodamico]

• I think option A would be best so the application code can remain agnostic to this.
• @asurak also flagged that the relay supports mTLS so the sidecar can ensure it only accepts traffic from the CF network. particularly to set up some (generous) rate limit rules.
• if the sidecar is within the same k8s (or equivalent) cluster, terminating TLS at the sidecar is reasonable.

[alessandromazza98]

• I think option A would be best so the application code can remain agnostic to this.
• @asurak also flagged that the relay supports mTLS so the sidecar can ensure it only accepts traffic from the CF network. particularly to set up some (generous) rate limit rules.
• if the sidecar is within the same k8s (or equivalent) cluster, terminating TLS at the sidecar is reasonable.

nice, is the infra team handling it? I think they may be the best for it as it's mostly an infra thing. You need to add one more service (the cloudflare sidecar) and put it in between cloudflare relay and our services

[asurak]

For most deployments we prefer if you don't handle TLS yourself and rather use AWS load balancer that will manage certificates, terminate TLS and deal with mTLS too. Unless you have a specific requirement why wouldn't want to use it. Considering how the data is handled here, I don't see a downside using standard design we have and going with AWS ALB/NLB.

Going with the Option A it would look like this. With a disclaimer that there would be N instances of the OHTTP Gateway.

flowchart LR
    cfog["Cloudflare Go-OHTTP-Gateway"]
    cfpg["Cloudflare Privacy Gateway"]
    awslb["AWS ALB/NLB"]
    Client -.->|OHTTP| cfog;
    Client -->|HTTPS| cfpg;
    cfpg -->|HTTPS| awslb;
    subgraph "AWS VPC"
    awslb -->|HTTP| cfog;
    subgraph "EKS cluster"
    cfog -->|HTTP| Gateway;
    cfog -->|HTTP| Indexer;
    end
end

Loading

This would make it a fairly standard design. Wdyt?

[alessandromazza98]

This would make it a fairly standard design. Wdyt?

seems very good to me

[paolodamico]

@alessandromazza98 should this be re-assigned?