DNS resolution issue with Asp.Net Alpine base image

[Suhumar]

When using mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.17-amd64 as the base image for the container, DNS resolution to payuk.api.virginmoney.com times out. It works sometimes but most of the time it times out.

By switching to mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim-amd64 the problem goes away.

Steps to Reproduce

This can be reproduced by creating a basic weatherforecast sample application and having this endpoint. Call to this endpoint will fail.

[HttpGet]
[Route("virgin")]
public async Task GetVirgin()
{
var result = await _httpClient.GetAsync("https://payuk.api.virginmoney.com/v2.0/.well-known/openid-configuration");
var payload = await result.Content.ReadAsStringAsync();
return payload;
}

Other Information

We found this post talking about the same issue - https://social.msdn.microsoft.com/Forums/vstudio/en-US/1acf650b-35b5-49e7-bd44-ca10f79f714a/dns-resolution-is-not-working-in-azure-web-app-for-containers-alpine-linux?forum=windowsazurewebsitespreview

We raised a support ticket in Azure and had a couple of Microsoft engineers look into this issue. As this became apparent this was a base image issue, they recommended to create a bug in Github.

Support request ID - 2211210050002481

Also this is not specific to Azure, we were able to reproduce this locally.

Because this is a DNS issue and timing out thought will provide the location as well. London (UK)

As part of the debug process, we changed the DNS server in Azure App Service to point to Google Name server and that also resolved the DNS issue. I changed the DNS server using the below line of code in the docker file

ENTRYPOINT ["/bin/sh" , "-c", "echo nameserver 8.8.8.8 >> /etc/resolv.conf && echo nameserver 8.8.4.4 >> /etc/resolv.conf && dotnet ********"]

[Suhumar]

This is the error I am seeing.

[mthalman]

Is this a regression for you or is this new development work that caused you to encounter this? If it's a regression, what was the configuration that was working? Or are you just attempting to migrate from Debian to Alpine and ran into this?

FWIW, I'm not able to repro, presumably because I've got a different DNS.

I'm curious if it works if you target Alpine 3.16 instead: mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.16-amd64.

You may want to take a look at this too: https://unix.stackexchange.com/questions/441664/alpine-linux-sometimes-dns-is-not-resolved

[Suhumar]

This is a new development and we started with Alpine image. Alpine 3.16 seems to have the same issue.

We did go down the route of adding google name server and that fixed the problem. But then that created other issues like our APIM could not be resolved on load. We saw a couple during the day and we had to revert back the change

[richlander]

Could you try a different base image for a bit while you try to determine the cause of the issue?

[JamiePed]

We saw this issue using Alpine images and we were able to fix the issue by moving to Debian images.

We were calling out to a service, who then updated their nameservers. They were updating from Google nameservers to Cloudflare.

When they made there change our services started timing out on DNS resolution. The error persisted until we pushed a new deployment using Debian as the new base image. (or they rolled back the change)

[mthalman]

@Suhumar - Have you tried running a curl or wget command on the URL? It'd be useful to isolate the repro. If you can determine that the issue still occurs with either of those commands, then this is an issue that stems from Alpine itself and not .NET's implementation.

[tri-bao]

It has been happening to us since 2 days on bitbucket pipeline which runs on AWS EC2 environment. It was fine 3 days ago (now is Feb 24th, 2023). It was unable to resolve domain api.nuget.api.

Running getent hosts api.nuget.org returned nothing

Running nslookup api.nuget.org got the following output. As we can see, no IP was resolved

Server:		10.61.128.2
Address:	10.61.128.2:53

Non-authoritative answer:
api.nuget.org	canonical name = nugetapiprod.trafficmanager.net
nugetapiprod.trafficmanager.net	canonical name = apiprod-mscdn.azureedge.net
apiprod-mscdn.azureedge.net	canonical name = apiprod-mscdn.afd.azureedge.net
apiprod-mscdn.afd.azureedge.net	canonical name = star-azureedge-prod.trafficmanager.net
star-azureedge-prod.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net
shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net	canonical name = global-entry-afdthirdparty-fallback-first.trafficmanager.net
global-entry-afdthirdparty-fallback-first.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fb-t-msedge.net
shed.dual-low.part-0012.t-0009.fb-t-msedge.net	canonical name = part-0012.t-0009.fb-t-msedge.net

Non-authoritative answer:
api.nuget.org	canonical name = nugetapiprod.trafficmanager.net
nugetapiprod.trafficmanager.net	canonical name = apiprod-mscdn.azureedge.net
apiprod-mscdn.azureedge.net	canonical name = apiprod-mscdn.afd.azureedge.net
apiprod-mscdn.afd.azureedge.net	canonical name = star-azureedge-prod.trafficmanager.net
star-azureedge-prod.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net
shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net	canonical name = global-entry-afdthirdparty-fallback-first.trafficmanager.net
global-entry-afdthirdparty-fallback-first.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fb-t-msedge.net
shed.dual-low.part-0012.t-0009.fb-t-msedge.net	canonical name = part-0012.t-0009.fb-t-msedge.net

But running nslookup api.nuget.org 8.8.8.8, the address resolved:

Server:		8.8.8.8
Address:	8.8.8.8:53

Non-authoritative answer:
api.nuget.org	canonical name = nugetapiprod.trafficmanager.net
nugetapiprod.trafficmanager.net	canonical name = apiprod-mscdn.azureedge.net
apiprod-mscdn.azureedge.net	canonical name = apiprod-mscdn.afd.azureedge.net
apiprod-mscdn.afd.azureedge.net	canonical name = star-azureedge-prod.trafficmanager.net
star-azureedge-prod.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net
shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net	canonical name = global-entry-afdthirdparty-fallback-first.trafficmanager.net
global-entry-afdthirdparty-fallback-first.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fb-t-msedge.net
shed.dual-low.part-0012.t-0009.fb-t-msedge.net	canonical name = part-0012.t-0009.fb-t-msedge.net
Name:	part-0012.t-0009.fb-t-msedge.net
Address: 13.107.253.40
Name:	part-0012.t-0009.fb-t-msedge.net
Address: 13.107.226.40

Non-authoritative answer:
api.nuget.org	canonical name = nugetapiprod.trafficmanager.net
nugetapiprod.trafficmanager.net	canonical name = apiprod-mscdn.azureedge.net
apiprod-mscdn.azureedge.net	canonical name = apiprod-mscdn.afd.azureedge.net
apiprod-mscdn.afd.azureedge.net	canonical name = star-azureedge-prod.trafficmanager.net
star-azureedge-prod.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net
shed.dual-low.part-0012.t-0009.fdv2-t-msedge.net	canonical name = global-entry-afdthirdparty-fallback-first.trafficmanager.net
global-entry-afdthirdparty-fallback-first.trafficmanager.net	canonical name = shed.dual-low.part-0012.t-0009.fb-t-msedge.net
shed.dual-low.part-0012.t-0009.fb-t-msedge.net	canonical name = part-0012.t-0009.fb-t-msedge.net
Name:	part-0012.t-0009.fb-t-msedge.net
Address: 2620:1ec:29:1::40
Name:	part-0012.t-0009.fb-t-msedge.net
Address: 2620:1ec:48:1::40

Could it be due to the IPv6?

We will be using the image tag sdk:6.0-alpine-amd64. It was cached by bitbucket docker cache. We don't know which sha256 digest was it.

When we use more specific version tag sdk:6.0.406-alpine3.17-amd64, It is working again.

---UPDATE---
I am not sure using the specific image tag like above will 100% work. It still failed for us but less sporadically!

Hope this helps

[bayanlouzi]

Hello @mthalman do you have any update can share on this issue ?

[mthalman]

See my comment here: #4382 (reply in thread)

[LukaszRacon]

This issue might be related to musl not supporting DNS-over-TCP.
See for more information:
https://betterprogramming.pub/why-i-will-never-use-alpine-linux-ever-again-a324fd0cbfd6
https://purplecarrot.co.uk/post/2021-09-04-does_alpine-resolve_dns_properly/
https://serverfault.com/questions/404840/when-do-dns-queries-use-tcp-instead-of-udp/404843#404843