Parallel DNS lookups sometimes fail

[dylanlan]

Version

v18.14.2

Platform

Darwin N61M66G4LW.lan 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Subsystem

dns

What steps will reproduce the bug?

Running multiple parallel dns.lookup() on my Apple Silicon Macbook sometimes gets the classic getaddrinfo ENOTFOUND error. It seems very similar to nodejs/node#28292

I first ran into this issue trying to use certain packages in parallel (eg: @aws-sdk/client-ssm). But then I found I could reproduce the error just doing dns lookups directly.

Sample script:

const dns = require('node:dns');
const util = require('util');

const promiseLookup = util.promisify(dns.lookup);

const host = 'github.com';
// const host = 'google.com'; // Certain hosts seems to succeed

async function test() {
    try {
        const requests = [];
        for (let i = 0; i < 1000; i++) {
            const options = {};
            // options.family = 4; // Specifying family: 4 seems to succeed
            // await promiseLookup(host, options); // Awaiting the lookups in sequence seems to succeed
            requests.push(promiseLookup(host, options));
        }
        await Promise.all(requests);
        console.log('success');
    } catch (err) {
        console.log(err);
    }
}

test();

I commented out 3 lines which each individually seem to fix the issue for me, but I'm not sure how to fix it for the package dependencies that I'm using.

How often does it reproduce? Is there a required condition?

I can reproduce it around 50% of the time with the sample script.

But it seems to reliably get fixed for me if any of these are true:

1. Running the dns.lookup() in sequence
2. Using { family: 4 } for the options argument
3. Looking up certain hosts (eg: google.com)
4. Using other OS (eg: Windows desktop, Ubuntu AWS EC2 instance, Intel processor Macbook)

Here are all the other things I've tried so far, that haven't seemed to fix it:

• Restarting my Macbook
• Flushing DNS cache with sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
• Upgrading to latest node 18.18.0 or 20.7.0
• Downgrading to node 16.18.1 (previous version I used)
• Disabling IPv6 (System Settings -> Network -> TCP/IP -> Configure IPv6, set to Link-Local Only)
• Disconnecting from my VPN
• Using only Ethernet
• Using only Wifi
• Disabling firewall and antivirus
• Using --dns-result-order=ipv4first or NODE_OPTIONS=--dns-result-order=ipv4first
• Using require('node:dns/promises')
• Changing configured DNS server from Mac default to Google's 8.8.8.8 or Cloudflare's 1.1.1.1

What is the expected behavior? Why is that the expected behavior?

The DNS lookups should consistently succeed when ran in parallel

What do you see instead?

Error: getaddrinfo ENOTFOUND github.com

> node sample-script.js
Error: getaddrinfo ENOTFOUND github.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'github.com'
}

Additional information

I haven't been able to reproduce it with parallel DNS lookups in a similar golang sample program on the same machine. I also haven't been able to reproduce it using dig, nslookup, or ping.

I've been using an Apple Silicon Macbook since 2021, but only started seeing this error during summer 2023. But I wasn't previously testing parallel DNS lookups, so I'm not sure if it's been an issue for me for 2 years or not.

My coworkers on other Apple Silicon Macbooks can also reproduce the issue. Interestingly, some of them on Intel processor Macbooks cannot seem to reproduce it.

I understand that there are potentially many similar issues and workarounds from nodejs/node#5436, and that it might be an OS-specific issue related to libc versions.

I'm confused why I'm not seeing any other recent issues for this other than from 2019 - maybe I'm missing something.

I'm wondering if there's any workarounds from node that I can use in the meantime. Since setting { family: 4 } seems to fix it for me, is there a way to globally set that setting in the node process? Otherwise I'm not sure how to cleanly override that setting when the DNS lookup is happening from nested package dependencies.

[aduh95]

@nodejs/dns

[bnoordhuis]

Inactionable as a bug report, the linked issue explains why it's not under node's control. Use dns.resolve() if dns.lookup() is giving you trouble.

Setting {family:4} probably means apple's libc uses one file descriptor less per lookup since it's not querying A and AAAA records in parallel.

I'll convert this to a discussion.

[dylanlan]

My goal is to find a suitable workaround for the specific package dependencies that I'm using. In this case, various @aws-sdk packages (eg: @aws-sdk/client-ssm).

Since it's a nested dependency package that is doing the DNS lookup, I cannot easily change it to use dns.resolve(), nor directly configure it to use {family:4}

My best workaround so far is to try and patch the dns.lookup method to always set family:4, but I'm not sure if this can cause other issues.

eg: something like this near process start:

const dns = require('node:dns');
const originalLookup = dns.lookup;

dns.lookup = function (domain, options, callback) {
    // Check if options are provided
    if (typeof options === 'function') {
        callback = options;
        options = {};
    }

    // Set family to 4
    options.family = 4;

    return originalLookup(domain, options, callback);
}

I plan to also reach out to AWS and Apple support, to see if they have any better workarounds. I'll add another comment if I find anything better

[dylanlan]

I had filed a similar issue for the AWS SDK: aws/aws-sdk-js-v3#5236

A couple other workarounds were suggested:

• Inject a different http handler that might not use dns.lookup or might cache the results better
• Add custom retry handler code anywhere that a dns.lookup happens

I also tried reaching out to Apple support, but wasn't successful.

I'm dealing with a fairly large codebase, so I plan to use my dns.lookup patch workaround above for now, instead of trying to update the code everywhere it's being used. Hopefully Apple will fix their DNS lookups at some point

[millerick]

I have similar problems in some of the codebases I work in. Setting UV_THREADPOOL_SIZE=200 in the environment before invoking node appears to gloss over the issue for me.

[danielvy]

I have a similar issue with @aws-sdk/client-s3 which led me to do some digging (also posted my findings to aws/aws-sdk-js-v3#5236).

I think that the reason this issue is not more wide spread is that, from what people report, it only happens under certain conditions:

1. Apple's silicone (not sure if it's only M1 or M2 too).
2. DNS records with short TTL (under 50s).
3. Bursts of parallel dns.loookup() calls.

Based on reports that increasing UV_THREADPOOL_SIZE may workaround the issue, it seems this is a concurrency edge case issue with Apple's implementation of getaddrinfo or how it's being called by libuv. I wouldn't hold my breath for Apple fixing this, this problem exists for quite some time.

I'd also be careful with setting UV_THREADPOOL_SIZE to a large value as it effects other API calls (fs, zlib). Curiously, setting UV_THREADPOOL_SIZE to 2 also makes the error go away (but higher values still trigger the error, albeit effecting the time until it happens the first time).

For new code, I'm wrapping my @aws-sdk command sending code with try/catch/retry logic. But this may be indeed tedious for existing large code base, so unfortunately, monkey patching dns.lookup() may be the best bet.

When patching dns.lookup(), it may be better to add some DNS caching logic (which Node doesn't do) instead of forcing IPv4. It may also have some performance benefits with high request volume (less calls to getaddrinfo, less actual DNS queries for short TTL). There was an existing DNS cache library that that did this, yahoo/dnscache, but it seems to be archived now (exploring the source code may be useful still).