Started getting random error inside docker container

[RollsChris]

Describe the Bug

When running our application inside a container, which attempts to connect to an azure service (keyVault for e.g.) we are getting the error: System.Net.Http.HttpRequestException: 'Name or service not known'

The exception is raised when we attempt to connect to the key vault, but I suspect its any connection.

Running outside the container everything is fine ( I have double checked all the enviroment variables).

Steps to Reproduce

we use a docker compose to run a set of services, when they start up they are unable to connect to the key vault using the code below ( pretty standard i think) .

` var keyVaultEndpoint = builtConfig["VAULTURI"];
var tenantId = builtConfig["AZUREAD:TENANTID"];
var clientId = builtConfig["AZUREAD:CLIENTID"];
var clientSecret = builtConfig["AZUREAD:CLIENTSECRET"];

                    if (!String.IsNullOrEmpty(keyVaultEndpoint)
                        && !String.IsNullOrEmpty(tenantId)
                        && !String.IsNullOrEmpty(clientId)
                        && !String.IsNullOrEmpty(clientSecret))
                    {
                        config.AddAzureKeyVault(
                            new AzureKeyVaultConfigurationOptions(
                                keyVaultEndpoint,
                                clientId,
                                clientSecret)
                            {
                                ReloadInterval = TimeSpan.FromSeconds(30),
                            }
                        );
                    }`

Other Information

Output of docker version

Docker version 20.10.8, build 3967b7d

Output of docker info

Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
compose: Docker Compose (Docker Inc., v2.0.0-rc.3)
scan: Docker Scan (Docker Inc., v0.8.0)

Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 4
Server Version: 20.10.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.104-microsoft-standard
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 24.98GiB
Name: docker-desktop
ID: WXP3:A6SG:FBGJ:QNC7:6PA3:C6EP:SQ5E:AMAQ:QBAD:O7ZR:QFPA:KWKM
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[MichaelSimons]

@karelz - Do you have any tips for how to diagnose and trace down this?

[karelz]

I wonder if it is problem with IPv6 not being enabled properly (based on WARNING: bridge-nf-call-ip6tables is disabled above) ... that may cause troubles (cc @antonfirsov).

I would recommend to try to hit the endpoint (KeyVault) from a simple HelloWorld-style console app, using HttpClient. You can then try its IP (IPv4 and/or IPv6).

If it is IPv6 problems, we introduced option to disable it in .NET 6. I am not sure which .NET version you're using though ...

[antonfirsov]

I can't confirm or deny this based on the info in the opening comment. Would be helpful to see a stack trace.

I wonder if it is problem with IPv6 not being enabled properly (based on WARNING: bridge-nf-call-ip6tables is disabled above) ...

bridge-nf-call-iptables is also disabled. Not familiar enough with Linux routing/bridging, but AFAIK ip(6)tables is just a packet filter (firewall) mechanism.

I would recommend to try to hit the endpoint (KeyVault) from a simple HelloWorld-style console app, using HttpClient.

Agreed. Would be nice to see how a small console app reacts to these endpoints (with stack trace).

[RollsChris]

Hi sorry for the late reply.. I have tested the key vault in a console app inside a docker container and everything works fine. The only difference is.. in azure we run in kubernetes..

[antonfirsov]

Can you try a console app in the Azure + kubernetes environment you are using? (Maybe a test copy of the production environment?) I understand this might be tricky to achieve, but it's not possible to diagnose this issue without more information.

[RollsChris]

Yea i understand its a weird one.. the error message isn't very useful. I will try and do it as soon as possible.

[RollsChris]

Hi so I have an update. Building and running the docker file works.. it breaks when we use docker-compose.
I have tested with this simple docker-compose file and it fails:

version: '3'
services:
    customermanagement.api:
      build:
        context: ..\..\..
        dockerfile: Services/CustomerManagement/CustomerManagement.Api/Dockerfile

here is the error

[KEYVAULTVALUE]testing
Unhandled exception. System.Net.Http.HttpRequestException: Name or service not known

 ---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known

   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

   --- End of inner exception stack trace ---

I dont think its anything to do with the key vault. it seems to be anything making comms out of the container when its been run with docker-compose.yml

Here is the code it fails on ( i added the last part for testing, please read comment)

 if (!String.IsNullOrEmpty(keyVaultUrl)
                            && !String.IsNullOrEmpty(tenantId)
                            && !String.IsNullOrEmpty(clientId)
                            && !String.IsNullOrEmpty(clientSecret))
                        {
                            config.AddAzureKeyVault(keyVaultUrl, clientId, clientSecret);
                        }

                        //Works, but not when code above is commented out 
                        var client = new SecretClient(new Uri(keyVaultUrl), new EnvironmentCredential());
                        var secret = client.GetSecretAsync("testSecret").Result;
                        var value = secret.Value.Value;
                        Console.WriteLine("[KEYVAULTVALUE]" + value);

Docker-compose version:
docker-compose version 1.29.2, build 5becea4c
Thanks