[omdb] Find internal DNS servers via /etc/resolv.conf

[jgallagher]

There are three changes in this PR:

1. If not given an explicit DNS server address, omdb now uses the system resolver instead of assuming a hardcoded rack subnet.
2. When sled-agent reconfigures the switch zone after getting an underlay address from RSS, it also configures and enables the network/dns/install SMF service to populate /etc/resolv.conf.
3. When sled-agent starts the switch zone on a cold boot after rack setup, it includes network/dns/install in the site manifest.

I tested changes 1 and 2 in a4x2, and things look good:

root@g0:~# grep 'enabled DNS' $(svcs -L sled-agent) | looker
22:10:59.677Z INFO SledAgent (ServiceManager): enabled DNS install service with new nameservers
    file = sled-agent/src/services.rs:4018
    nameservers = [fd00:17:1:1::1, fd00:17:1:2::1, fd00:17:1:3::1]

root@oxz_switch:~# cat /etc/resolv.conf
nameserver fd00:17:1:1::1
nameserver fd00:17:1:2::1
nameserver fd00:17:1:3::1
root@oxz_switch:~# omdb nexus blueprints list
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:17:1:d01::5]:12232
...

I'll need a racklette to test change 3; I'll wait until #9653 lands to do so. (Or I'll rebase on top of it to test early next week, if it hasn't landed yet.)

Fixes #9606.

[smklein]

maybe a dumb question - but if we tried to change the rack subnet after initialization, we'd presumably be taking this path?

I know the comment here is describing "moving from the bootstrap to underlay network", which is different - but I just want to know when/how the updated resolv.conf would get populated.

[jgallagher]

I believe that's correct.

[jgallagher]

When sled-agent starts the switch zone on a cold boot after rack setup, it includes network/dns/install in the site manifest.

I tested this on dublin today. Cold booting sled 16, we see sled-agent configure the DNS install service up front:

00:02:03.584Z INFO SledAgent (ServiceManager): enabled DNS install service with new nameservers
    file = sled-agent/src/services.rs:4016
    nameservers = [fde5:b59c:e880:1::1, fde5:b59c:e880:2::1, fde5:b59c:e880:3::1]

and /etc/resolv.conf is populated within the switch zone:

root@oxz_switch1:~# cat /etc/resolv.conf
nameserver fde5:b59c:e880:1::1
nameserver fde5:b59c:e880:2::1
nameserver fde5:b59c:e880:3::1
root@oxz_switch1:~# omdb db sleds
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fde5:b59c:e880:102::3]:32221,[fde5:b59c:e880:104::3]:32221,[fde5:b59c:e880:104::4]:32221,[fde5:b59c:e880:103::3]:32221,[fde5:b59c:e880:101::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (220.0.0)
note: listing all commissioned sleds (use -F to filter, e.g. -F in-service)
 SERIAL       IP                             ROLE      POLICY      STATE   ID
 BRM23230018  [fde5:b59c:e880:103::1]:12345  scrimlet  in service  active  4412e4e3-4bea-463c-8556-7fd813f9720b
 BRM42220026  [fde5:b59c:e880:101::1]:12345  scrimlet  in service  active  56918b53-c015-411e-94bf-9cbc3badc293
 BRM23230010  [fde5:b59c:e880:104::1]:12345  -         in service  active  b186935b-141c-4fe5-8905-121137c6c051
 BRM27230037  [fde5:b59c:e880:102::1]:12345  -         in service  active  f17fdf7b-4a53-4446-8fdf-b1d25a02792f

(1 and 2 also worked on dublin as they did in a4x2.)