valet install keeps asking for system certificates permissions on MacOS 11 Big Sur

[nickdekruijk]

Clear description of your problem

When running valet install I keep getting popups from MacOS asking for permission to change system certificates. Even if I agree it keeps popping up until I cancel te install script.

Expected behavior

Don't ask for permission or maybe just once or twice.

Current behavior

Looks like an infinite loop of permission requirement popups

Steps to Reproduce

1. Clean Big Sur installation on Macbook Pro
2. Follow steps on https://laravel.com/docs/8.x/valet#installation

Output of these steps

Stopping nginx...
Installing nginx...
[nginx] is not installed, installing it now via Brew... 🍻
Installing nginx configuration...
Installing nginx directory...

Then the popup loop starts

Possible solution

I noticed brew giving a notification about not being fully compatible with Big Sur yet. Not sure if that's related.

Diagnosis

sw_vers

ProductName:	macOS
ProductVersion:	11.0.1
BuildVersion:	20B29

valet --version

Laravel Valet 2.13.0

cat ~/.config/valet/config.json

{
    "tld": "test",
    "paths": [
        "/Users/nick/.config/valet/Sites",
        "/Users/nick/Documents/Klanten"
    ]
}

cat ~/.composer/composer.json

{
    "require": {
        "laravel/valet": "^2.13"
    }
}

composer global diagnose

Changed current directory to /Users/nick/.composer
Checking composer.json: WARNING
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys: FAIL
Missing pubkey for tags verification
Missing pubkey for dev verification
Run composer self-update --update-keys to set them up
Checking composer version: OK
Composer version: 2.0.6
PHP version: 7.4.12
PHP binary path: /usr/local/Cellar/php/7.4.12/bin/php
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020
cURL version: 7.73.0 libz 1.2.11 ssl OpenSSL/1.1.1h
zip extension: OK

composer global outdated

Changed current directory to /Users/nick/.composer

ls -al /etc/sudoers.d/

total 0
drwxr-xr-x   2 root  wheel    64  1 jan  2020 .
drwxr-xr-x  85 root  wheel  2720 13 nov 12:22 ..

brew config

HOMEBREW_VERSION: 2.5.10
ORIGIN: https://github.com/Homebrew/brew
HEAD: b43c0fed789b4cae33cd200284c44a095db57c3c
Last commit: 17 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: c650db22d9f3d08d6a03fccc0c1f9098fea4993b
Core tap last commit: 2 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit kabylake
Clang: 12.0 build 1200
Git: 2.24.3 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 11.0.1-x86_64
CLT: 12.2.0.0.1.1604076827
Xcode: N/A

brew services list

Name  Status  User Plist
nginx stopped      
php   stopped

brew list --versions | grep -E "(php|nginx|dnsmasq|mariadb|mysql|mailhog|openssl)(@\d\..*)?\s"

curl-openssl 7.73.0
nginx 1.19.4
openssl@1.1 1.1.1h
php 7.4.12

brew outdated

php -v

PHP 7.4.12 (cli) (built: Oct 29 2020 18:37:21) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.12, Copyright (c), by Zend Technologies

which -a php

/usr/local/bin/php
/usr/bin/php

php --ini

Configuration File (php.ini) Path: /usr/local/etc/php/7.4
Loaded Configuration File:         /usr/local/etc/php/7.4/php.ini
Scan for additional .ini files in: /usr/local/etc/php/7.4/conf.d
Additional .ini files parsed:      /usr/local/etc/php/7.4/conf.d/ext-opcache.ini

nginx -v

nginx version: nginx/1.19.4

curl --version

curl 7.64.1 (x86_64-apple-darwin20.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

php --ri curl

curl
cURL support => enabled

cURL Information => 7.73.0

Age => 7

Features

AsynchDNS => Yes

CharConv => No

Debug => No

GSS-Negotiate => No

IDN => No

IPv6 => Yes

krb4 => No

Largefile => Yes

libz => Yes

NTLM => Yes

NTLMWB => Yes

SPNEGO => Yes

SSL => Yes

SSPI => No

TLS-SRP => Yes

HTTP2 => Yes

GSSAPI => Yes

KERBEROS5 => Yes

UNIX_SOCKETS => Yes

PSL => No

HTTPS_PROXY => Yes

MULTI_SSL => No

BROTLI => Yes

Protocols => dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp, rtsp, scp, sftp, smb, smbs, smtp, smtps, telnet, tftp

Host => x86_64-apple-darwin19.6.0

SSL Version => OpenSSL/1.1.1h

ZLib Version => 1.2.11

libSSH Version => libssh2/1.9.0
Directive => Local Value => Master Value

curl.cainfo => no value => no value

~/.composer/vendor/laravel/valet/bin/ngrok version

ngrok version 2.3.35

ls -al ~/.ngrok2

ls: /Users/nick/.ngrok2: No such file or directory

brew info nginx

nginx: stable 1.19.4 (bottled), HEAD
HTTP(S) server and reverse proxy, and IMAP/POP3 proxy server
https://nginx.org/
/usr/local/Cellar/nginx/1.19.4 (25 files, 2.2MB) *
  Poured from bottle on 2020-11-13 at 14:21:43
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/nginx.rb
License: BSD-2-Clause
==> Dependencies
Required: openssl@1.1, pcre
==> Options
--HEAD
	Install HEAD version
==> Caveats
Docroot is: /usr/local/var/www
The default port has been set in /usr/local/etc/nginx/nginx.conf to 8080 so that

nginx can run without sudo.
nginx will load all files in /usr/local/etc/nginx/servers/.
To have launchd start nginx now and restart at login:

brew services start nginx

Or, if you don't want/need a background service you can just run:

nginx

==> Analytics

install: 47,928 (30 days), 115,204 (90 days), 434,260 (365 days)

install-on-request: 47,219 (30 days), 113,224 (90 days), 422,345 (365 days)

build-error: 0 (30 days)

brew info php

php: stable 7.4.12 (bottled), HEAD
General-purpose scripting language
https://www.php.net/
/usr/local/Cellar/php/7.4.12 (497 files, 72.3MB) *
  Poured from bottle on 2020-11-13 at 14:11:55
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/php.rb
License: PHP-3.01
==> Dependencies
Build: httpd, pkg-config
Required: apr, apr-util, argon2, aspell, autoconf, curl-openssl, freetds, gd, gettext, glib, gmp, icu4c, krb5, libffi, libpq, libsodium, libzip, oniguruma, openldap, openssl@1.1, pcre2, sqlite, tidy-html5, unixodbc
==> Options
--HEAD
	Install HEAD version
==> Caveats
To enable PHP in Apache add the following to httpd.conf and restart Apache:
    LoadModule php7_module /usr/local/opt/php/lib/httpd/modules/libphp7.so
<FilesMatch \.php$>
    SetHandler application/x-httpd-php
</FilesMatch>

Finally, check DirectoryIndex includes index.php

DirectoryIndex index.php index.html
The php.ini and php-fpm.ini file can be found in:

/usr/local/etc/php/7.4/
To have launchd start php now and restart at login:

brew services start php

Or, if you don't want/need a background service you can just run:

php-fpm

==> Analytics

install: 55,181 (30 days), 155,907 (90 days), 584,143 (365 days)

install-on-request: 54,226 (30 days), 152,798 (90 days), 556,269 (365 days)

build-error: 0 (30 days)

brew info openssl

openssl@1.1: stable 1.1.1h (bottled) [keg-only]
Cryptography and SSL/TLS Toolkit
https://openssl.org/
/usr/local/Cellar/openssl@1.1/1.1.1h (8,067 files, 18.5MB)
  Poured from bottle on 2020-11-13 at 14:10:12
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/openssl@1.1.rb
License: OpenSSL
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /usr/local/etc/openssl@1.1/certs
and run

/usr/local/opt/openssl@1.1/bin/c_rehash
openssl@1.1 is keg-only, which means it was not symlinked into /usr/local,

because macOS provides LibreSSL.
If you need to have openssl@1.1 first in your PATH run:

echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc
For compilers to find openssl@1.1 you may need to set:

export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"

export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"
==> Analytics

install: 824,226 (30 days), 1,888,039 (90 days), 7,298,419 (365 days)

install-on-request: 134,646 (30 days), 272,362 (90 days), 1,031,059 (365 days)

build-error: 0 (30 days)

openssl version -a

LibreSSL 2.8.3
built on: date not available
platform: information not available
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: information not available
OPENSSLDIR: "/private/etc/ssl"

openssl ciphers

ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:GOST2012256-GOST89-GOST89:DHE-RSA-CAMELLIA256-SHA256:DHE-RSA-CAMELLIA256-SHA:GOST2001-GOST89-GOST89:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA256:DHE-RSA-CAMELLIA128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-SHA256:CAMELLIA128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA

sudo nginx -t

nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

which -a php-fpm

/usr/local/sbin/php-fpm
/usr/sbin/php-fpm

/usr/local/opt/php/sbin/php-fpm -v

PHP 7.4.12 (fpm-fcgi) (built: Oct 29 2020 18:37:31)
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.12, Copyright (c), by Zend Technologies

sudo /usr/local/opt/php/sbin/php-fpm -y /usr/local/etc/php/7.4/php-fpm.conf --test

[13-Nov-2020 14:31:33] NOTICE: configuration file /usr/local/etc/php/7.4/php-fpm.conf test is successful

ls -al ~/Library/LaunchAgents | grep homebrew

ls: /Users/nick/Library/LaunchAgents: No such file or directory

ls -al /Library/LaunchAgents | grep homebrew

ls -al /Library/LaunchDaemons | grep homebrew

[nickdekruijk]

Update: Seems it's not an infinite loop, apparently valet tries to secure all my sites I had on my previous MacOs Catalina. I noticed keychain (search for 'valet') results growing each time I grant permission. So after a 100 times or so valet finally installed. No idea where valet found the 'old' config tho. Maybe iCloud documents syncing?

Manually running valet secure also triggers the same popup. Probably a new MacOS security feature, hopefully it can be avoided in a future valet update?

[drbyte]

Yes, MacOS syncs your keychain to iCloud.

valet's install, tld, secure, unsecure, and unsecure --all commands interact with certificates and attempt to keep local changes in sync with the Keychain.

If you are running into this "loop" of keychain prompts, it's probably because Valet is looping through all the Nginx site configs you've previously secured (it does this as part of install to ensure that all the Nginx configs are valid).

When switching Macs or wiping and reinstalling MacOS you will have a new Keychain for that Mac. MacOS will sync your keychain from iCloud so you'll have access to previous security data associated with your Apple ID. Keychain then attempts to be helpful by allowing you access to the "old" information but requires entering a password unless those details are in the currently unlocked keychain areas.

In the end it's going to require your password for each domain certificate your keychain has. Go through the pain of it once and things should be fine moving forward.

Best Option
I suppose you could run valet unsecure --all before switching macs and before doing your backups. This will remove all the (site) certificates from the keychain seamlessly.

[drbyte]

Below are some additional thoughts about what one could explore. They may or may not be helpful to your specific situation.

Manual Keychain Cleanup
You could run the Keychain Access application on your Mac and search for @laravel.valet to identify all certificates added by Valet. Then either move them to your new System keychain, or just delete them all (one by one, yes it's a slow process).
It's pretty complicated to automate this, since Keychain's password may not be linked to the current Mac's sudo credentials, and thus it requires manually keying the password to access that older data.
I'd suggest avoiding this step and just answer the password prompts during valet install instead, and tell Keychain to "Always Allow". On each one.

Other considerations
Running valet unsecure --all will attempt to delete certificates for all sites Valet knows about. It will delete the certificate files from your local filesystem and from the nginx configs in your local filesystem, and also delete those associated certificates from the Keychain.
First it will show you a list in the console of all the sites it is about to attempt to unsecure.
If you're getting prompts for Keychain password while doing that, then you can abort the process, but you'll have a list of sites to re-secure manually later after the following aggressive clearing.
(You can get almost the same list by running valet links, but it only shows linked sites, not parked sites.)

You could also delete the ~/.config/valet/CA folder from your Mac. Then run valet secure on any link/site. This will trigger the generation of a new Valet CA cert (because you deleted the old local cert file for it) which it will push to the current Keychain. (You must run valet secure in order to generate the new CA.)
Then you could run valet tld test to rebuild all secure sites. This is largely what valet install also triggers, so will probably also prompt for your Keychain password so it can update all the certs with matching domain names in your Keychain. But having already generated a new CA things may be easier going forward.

[exploriter]

Similar issue for me.

In the end it's going to require your password for each domain certificate your keychain has. Go through the pain of it once and things should be fine moving forward.

It prompts me for the password for every site every time I run valet install. There is no option to "Always Allow" on these prompts.

I've deleted from Keychain (although mine is not pulling from old configs—seems fine there), unsecured all, deleted the ~/.config/valet/CA folder, and gone through the secure process to no avail.

[drbyte]

@Gigawatson What's the history of this Mac? (Being unable to replicate, I need to know more of what's happened in order to troubleshoot.)

What MacOS version was on it previously?

Did you upgrade-in-place?
Or was it a complete wipe of the HD and install fresh? On a fresh OS, without restoring from any backups (but linking to your same AppleID), does the problem happen?

Does running valet uninstall --force, followed by a fresh install of Valet solve the issue? (remember to backup your ~/.config/valet directory if you've got something custom in there that you might want to recreate later)

[exploriter]

What's the history of this Mac?

Complete wipe ~ month ago; didn't restore from backup (but linked the same Apple ID).

Will try an uninstall with the --force flag and report back.

[SimJoSt]

I am running into the same issue. Even after unsecuring all sites, removing the CA folder, I had to enter the password again for every site I secured again as well as all following valet install commands.
I also deleted the Certificates folder, because valet install will not just register all sites visible by valet links, but all certificates including the ones from old sites.

Every time im am running an update on my computer (brew (php/composer), composer packages, etc.) I run valet install again to adjust the configuration. It's not feasible to enter the password for every certificate again.

Is there another solution?

[drbyte]

Does running valet tld test (or valet tld local followed by valet tld test just to change the tld to something completely different temporarily) make any difference? Changing the tld will rebuild all certificates for sites that have certificates created by valet secure.

[exploriter]

I did a valet uninstall --force and reinstalled, added back my sites, secured them, and tried a second valet install to see if it would prompt me... and it did.

For shits and giggles, I tried the valet tld local test and it made no difference.

[drbyte]

I've deleted from Keychain

What did you delete?

You shouldn't be getting system certificate permission prompts to "add" if the keychain already has your most current Valet CA.
And you shouldn't be getting prompts to access existing ones if the keychain doesn't have the certificate that valet is trying to access or or delete/replace.

[drbyte]

Is there certainty whether this is unique to Big Sur? (I'm not using it, so can't explore hands-on for that.)
Or is it just a byproduct of OS upgrades?
Or is it just for cases where iCloud keychains (with former Valet entries) are being auto-synced to a machine that hasn't used them before?

[exploriter]

What did you delete?

Everything related to Valet.

You shouldn't be...

I know, but I do. I've removed everything relating to Valet both on the computer and in iCloud. Yet, the problem remains. I don't know if this is strictly a Big Sur issue as I haven't used Valet in some time (although I do not remember this as an issue in previous OS versions).

[drbyte]

Can you provide screenshots of what MacOS is prompting you for?

Valet interacts with certs (for itself and for each site you secure), CAs, and sudo.
I've been making assumptions about what you're having to provide a password for, and I suspect we may be talking about different things. Specifics will help clarify.

[exploriter]

As you can see, I am prompted for my password 6 times which corresponds to the number of secured sites I have.

[drbyte]

Thanks. That's helpful.