How can I use token to login to a registry behind reverse proxy

[PaleNeutron]

My company have a harbor deployed at "https://dockerhub.comp.com" but I can not access it directly.

So I create a reverse proxy to it and I can access it at "http://amdockerhub.comp.com".

But the problem is when I try to login, podman will get login url from registry :

podman login amdockerhub.comp.com

I got

DEBU[0005] GET http://amdockerhub.comp.com/v2/    
DEBU[0005] Ping http://amdockerhub.comp.com/v2/ status 401 
DEBU[0005] GET https://dockerhub.comp.com/service/token?account=dap-am-framework&service=harbor-registry 

http://amdockerhub.comp.com/v2/ response contains header:

www-authenticate: Bearer realm="https://dockerhub.comp.com/service/token",service="harbor-registry"

So podman use this origin url to login and failed.

I can get token by curl but I don't know how to add it to podman.

[rhatdan]

@mtrmac @vrothberg PTAL

[mtrmac]

… guess… if you are interposing a proxy in front of the registry, that proxy could rewrite the WWW-Authenticate header as well?

Maybe? Forwarding token requests and responses though a proxy seems like something that authentication systems might want to actively protect against, see various “audience” fields in Open ID tokens and the like.

I think this protocol is simple / naive enough that just editing the headers, and forwarding bearer tokens, would work fine. But I have little experience with proxies of this kind.

[PaleNeutron]

@mtrmac Thanks!

I add this to my nginx.conf and it works.

  set $auth_header '';
  if ($upstream_http_www_authenticate = "") {
    set $auth_header 'Bearer realm="http://amdockerhub.comp.com/service/token",service="harbor-registry"';
  } 
  proxy_hide_header www-authenticate;
  add_header www-authenticate $auth_header always;

But I am still confused why podman use TLS first even this registry is marked as insecure, see logs:

DEBU[0000] Loading registries configuration "/home/john/.config/containers/registries.conf" 
DEBU[0000] No credentials for amdockerhub.comp.com found 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/amdockerhub.comp.com 
DEBU[0000] GET https://amdockerhub.comp.com/v2/   
DEBU[0005] Ping https://amdockerhub.comp.com/v2/ err Get "https://amdockerhub.comp.com/v2/": read tcp 192.168.115.234:60138->172.23.128.1:443: read: connection reset by peer (&url.Error{Op:"Get", URL:"https://amdockerhub.comp.com/v2/", Err:(*net.OpError)(0xc0005ce9b0)}) 
DEBU[0005] GET http://amdockerhub.comp.com/v2/    
DEBU[0005] Ping http://amdockerhub.comp.com/v2/ status 401 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/v2/    
DEBU[0005] Stored credentials for amdockerhub.comp.com in credential helper containers-auth.json 
Used:  /run/user/1000/containers/auth.json
Login Succeeded!

This also happens to push:

Podman try to use https somehow

DEBU[0005] PATCH https://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/339348b7-5673-4dca-97aa-2c4e1da9ab71?_state=OJZBVkU-9cTy2D7M5K77NdgVmn8nsdxL2UKRS-thy1d7Ik5hbWUiOiJkYXAt
Copying blob 0eeefc073e47 [--------------------------------------] 8.0b / 10.0KiB
Copying blob 8db562787e01 [--------------------------------------] 8.0b / 3.5KiB
Copying blob c8cf2918b5d1 [--------------------------------------] 8.0b / 45.8MiB
Copying blob 8fbc01148871 [--------------------------------------] 8.0b / 79.4MiB
Copying blob 950d1cd21157 [--------------------------------------] 8.0b / 75.2MiB

Full push log

$ podman push --tls-verify=false amdockerhub.comp.com/dap-am-framework/home-entry  --log-level=debug   
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman push --tls-verify=false amdockerhub.comp.com/dap-am-framework/home-entry --log-level=debug) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/etc/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/john/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/podman-run-1000/containers" from database 
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/john/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/podman-run-1000/containers 
DEBU[0000] Using static dir /home/john/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp 
DEBU[0000] Using volume path /home/john/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/john/.config/cni/net.d/87-podman.conflist 
INFO[0000] Found CNI network kibana_default (type=bridge) at /home/john/.config/cni/net.d/kibana_default.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman push --tls-verify=false amdockerhub.comp.com/dap-am-framework/home-entry --log-level=debug) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/etc/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/john/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/podman-run-1000/containers" from database 
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/john/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/podman-run-1000/containers 
DEBU[0000] Using static dir /home/john/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp 
DEBU[0000] Using volume path /home/john/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/john/.config/cni/net.d/87-podman.conflist 
INFO[0000] Found CNI network kibana_default (type=bridge) at /home/john/.config/cni/net.d/kibana_default.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Looking up image "amdockerhub.comp.com/dap-am-framework/home-entry" in local containers storage 
DEBU[0000] Trying "amdockerhub.comp.com/dap-am-framework/home-entry" ... 
DEBU[0000] Trying "amdockerhub.comp.com/dap-am-framework/home-entry:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/john/.local/share/containers/storage+/tmp/podman-run-1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@fcea483176cdade0b4a391163847d83ba10c1be4d3bf7c6826e0133e9232309b" 
DEBU[0000] Found image "amdockerhub.comp.com/dap-am-framework/home-entry" as "amdockerhub.comp.com/dap-am-framework/home-entry:latest" in local containers storage 
DEBU[0000] Found image "amdockerhub.comp.com/dap-am-framework/home-entry" as "amdockerhub.comp.com/dap-am-framework/home-entry:latest" in local containers storage ([overlay@/home/john/.local/share/containers/storage+/tmp/podman-run-1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@fcea483176cdade0b4a391163847d83ba10c1be4d3bf7c6826e0133e9232309b) 
DEBU[0000] Pushing image amdockerhub.comp.com/dap-am-framework/home-entry to amdockerhub.comp.com/dap-am-framework/home-entry 
DEBU[0000] Copying source image [overlay@/home/john/.local/share/containers/storage+/tmp/podman-run-1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@fcea483176cdade0b4a391163847d83ba10c1be4d3bf7c6826e0133e9232309b to destination image //amdockerhub.comp.com/dap-am-framework/home-entry:latest 
DEBU[0000] Loading registries configuration "/home/john/.config/containers/registries.conf" 
DEBU[0000] Found credentials for amdockerhub.comp.com in credential helper containers-auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  No signature storage configuration found for amdockerhub.comp.com/dap-am-framework/home-entry:latest, using built-in default file:///home/john/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/amdockerhub.comp.com 
DEBU[0000] Using blob info cache at /home/john/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/john/.local/share/containers/storage]@fcea483176cdade0b4a391163847d83ba10c1be4d3bf7c6826e0133e9232309b 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:c8cf2918b5d179e4fd674b00b23539890d106feaf593cf5bda4a9ed8b61d0acb 
DEBU[0000] GET https://amdockerhub.comp.com/v2/   
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:0eeefc073e4779a251aca6760f4cb755f205077a79b42343cd6e9d2e9eba178d 
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:8db562787e0175e5543eec0d922983a6101ce63aef67304500dec12eeb0a1542 
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:b70a62e9a5ce3e92bb70401e850126deb3b613a2d094f699e8d7eae88eb06fe5 
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:950d1cd211572857ec798f05767ad7614025ec505d4e85df9d5e62662ed4fea9 
DEBU[0000] Checking /v2/dap-am-framework/home-entry/blobs/sha256:8fbc0114887126b319002ff3bc0e446ae835936e88177d141376e013d90f0ec0 
DEBU[0005] Ping https://amdockerhub.comp.com/v2/ err Get "https://amdockerhub.comp.com/v2/": read tcp 192.168.115.234:39824->172.23.128.1:443: read: connection reset by peer (&url.Error{Op:"Get", URL:"https://amdockerhub.comp.com/v2/", Err:(*net.OpError)(0xc00101a000)}) 
DEBU[0005] GET http://amdockerhub.comp.com/v2/    
DEBU[0005] Ping http://amdockerhub.comp.com/v2/ status 401 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] GET http://amdockerhub.comp.com/service/token?account=dap-am-framework&scope=repository%3Adap-am-framework%2Fhome-entry%3Apull%2Cpush&service=harbor-registry 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:8fbc0114887126b319002ff3bc0e446ae835936e88177d141376e013d90f0ec0 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:0eeefc073e4779a251aca6760f4cb755f205077a79b42343cd6e9d2e9eba178d 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:c8cf2918b5d179e4fd674b00b23539890d106feaf593cf5bda4a9ed8b61d0acb 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:b70a62e9a5ce3e92bb70401e850126deb3b613a2d094f699e8d7eae88eb06fe5 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:8db562787e0175e5543eec0d922983a6101ce63aef67304500dec12eeb0a1542 
DEBU[0005] HEAD http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/sha256:950d1cd211572857ec798f05767ad7614025ec505d4e85df9d5e62662ed4fea9 
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "36b4e7520f0db4d24d4d127d50fab61d68220bc4bbb30c44729f1e75b3a17108" without compression for blob "sha256:8fbc0114887126b319002ff3bc0e446ae835936e88177d141376e013d90f0ec0" 
DEBU[0005] ... not present                              
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "d97a40e0157a69e954a5225314e05cfffa1bb09a0e51088701b5bb32059f0ca7" without compression for blob "sha256:0eeefc073e4779a251aca6760f4cb755f205077a79b42343cd6e9d2e9eba178d" 
DEBU[0005] exporting filesystem layer "4859c9ccf30fcc1c419598854ba6dd10939bb58ac0d0abf491b39485e342136e" without compression for blob "sha256:c8cf2918b5d179e4fd674b00b23539890d106feaf593cf5bda4a9ed8b61d0acb" 
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "6b95b2b4a8542251f2063fea2c8bfae9ecd6da3cef4c4cd6bb42ac31666e1cca" without compression for blob "sha256:b70a62e9a5ce3e92bb70401e850126deb3b613a2d094f699e8d7eae88eb06fe5" 
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "11c51ec99c7930d786cfa8bba957e9bfba953da11d11ca57468b6141f67ca4cd" without compression for blob "sha256:8db562787e0175e5543eec0d922983a6101ce63aef67304500dec12eeb0a1542" 
DEBU[0005] No compression detected                      
DEBU[0005] Compressing blob on the fly                  
DEBU[0005] Uploading /v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] POST http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] No compression detected                      
DEBU[0005] Compressing blob on the fly                  
DEBU[0005] Uploading /v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] POST http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] ... not present                              
DEBU[0005] exporting filesystem layer "950d1cd211572857ec798f05767ad7614025ec505d4e85df9d5e62662ed4fea9" without compression for blob "sha256:950d1cd211572857ec798f05767ad7614025ec505d4e85df9d5e62662ed4fea9" 
DEBU[0005] PATCH https://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/0ebc3b34-ff6a-494d-b484-3b682dd96c73?_state=KPobaCj8Ho5L9NzeuCRo9AFFwNq72uoYU5fG5Ub69nV7Ik5hbWUiOiJkYXAtYW0tZnJhbWV3b3JrL2hvbWUtZW50cnkiLCJVVUlEIjoiMGViYzNiMzQtZmY2YS00OTRkLWI0ODQtM2I2ODJkZDk2YzczIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA5LTIwVDAyOjUwOjM0Ljg3MjUzOTQ4N1oifQ%3D%3D 
DEBU[0005] PATCH https://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/412dd019-9455-4307-9134-9b89b9ce440d?_state=hcwVi0p_gPrj8Vbc6WCTOvXf4EYK64JGGE1j4Q6mQbt7Ik5hbWUiOiJkYXAtYW0tZnJhbWV3b3JrL2hvbWUtZW50cnkiLCJVVUlEIjoiNDEyZGQwMTktOTQ1NS00MzA3LTkxMzQtOWI4OWI5Y2U0NDBkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA5LTIwVDAyOjUwOjM0Ljg3MjI1MTEyMVoifQ%3D%3D 
DEBU[0005] No compression detected                      
DEBU[0005] Compressing blob on the fly                  
DEBU[0005] Uploading /v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] POST http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] PATCH https://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/4f359fef-eb85-4fa0-bc89-cc74def43d38?_state=_u-QZafKToCP2eDnKAbMh_O4Mb36BcwCx1T9tRTQxMl7Ik5hbWUiOiJkYXAtYW0tZnJhbWV3b3JrL2hvbWUtZW50cnkiLCJVVUlEIjoiNGYzNTlmZWYtZWI4NS00ZmEwLWJjODktY2M3NGRlZjQzZDM4IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA5LTIwVDAyOjUwOjM0Ljk0Njc5NzY4WiJ9 
Copying blob 0eeefc073e47 [--------------------------------------] 8.0b / 10.0KiB
Copying blob 8db562787e01 [--------------------------------------] 8.0b / 3.5KiB
Copying blob c8cf2918b5d1 [--------------------------------------] 8.0b / 45.8MiB
DEBU[0005] No compression detected                      
DEBU[0005] Compressing blob on the fly                  
DEBU[0005] Uploading /v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] POST http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] No compression detected                      
DEBU[0005] Compressing blob on the fly                  
DEBU[0005] Uploading /v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] POST http://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/ 
DEBU[0005] PATCH https://amdockerhub.comp.com/v2/dap-am-framework/home-entry/blobs/uploads/339348b7-5673-4dca-97aa-2c4e1da9ab71?_state=OJZBVkU-9cTy2D7M5K77NdgVmn8nsdxL2UKRS-thy1d7Ik5hbWUiOiJkYXAt
Copying blob 0eeefc073e47 [--------------------------------------] 8.0b / 10.0KiB
Copying blob 8db562787e01 [--------------------------------------] 8.0b / 3.5KiB
Copying blob c8cf2918b5d1 [--------------------------------------] 8.0b / 45.8MiB
Copying blob 8fbc01148871 [--------------------------------------] 8.0b / 79.4MiB
Copying blob 950d1cd21157 [--------------------------------------] 8.0b / 75.2MiB
DEBU[0006] No compression detected                      
Copying blob 0eeefc073e47 [--------------------------------------] 8.0b / 10.0KiB
Copying blob 8db562787e01 [--------------------------------------] 8.0b / 3.5KiB
Copying blob c8cf2918b5d1 [--------------------------------------] 8.0b / 45.8MiB
Copying blob 8fbc01148871 [--------------------------------------] 8.0b / 79.4MiB
Copying blob 950d1cd21157 [--------------------------------------] 8.0b / 75.2MiB
Copying blob b70a62e9a5ce [--------------------------------------] 8.0b / 359.4MiB
^CINFO[0009] Received shutdown signal interrupt, terminating! 
INFO[0009] Invoking shutdown handler libpod             
INFO[0009] Received shutdown signal interrupt, terminating! 
INFO[0009] Invoking shutdown handler libpod             

[vrothberg]

But I am still confused why podman use TLS first even this registry is marked as insecure

For security reasons. TLS may work despite being configured as insecure. So Podman will attempt TLS first and, iff configured as insecure, fallback to skipping TLS verification.

[PaleNeutron]

I can see in login workflow, it fallback to http, but in push work flow, it kept using https and finally fails. See full push log.

[PaleNeutron]

Could be resolved by adding proxy_redirect 'https://amdockerhub.comp.com' 'http://amdockerhub.comp.com'; in nginx.conf

[mtrmac]

TLS may work despite being configured as insecure.

Not … very usefully; with TLS verification disabled, we don’t verify the server’s certificate, so such a TLS protects against passive observation (assuming a forward-secure cipher suite) but never against an active attacker.

To an extent it’s a historical accident that “disable TLS security properties” and “try both port 443 with TLS and port 80 without” are a single setting. But we just can’t change that any longer, that would break existing users.

We could add two new options… but it’s hard to prioritize that (there has been at least one report that trying TLS against servers which don’t speak it can lead to delays, but just the one IIRC). Think also about RFC 7258… disabling TLS Should Not Be Done™️ outside of VMs on a local machine.

[mtrmac]

The push behavior is the same situation as with the token: a response contains a full URL, and a proxy for this API must rewrite that.