Test monitor fails to authenticate

[vivian-rook]

Deploying a test cluster with:

helm install core core --namespace neuvector --create-namespace --repo https://neuvector.github.io/neuvector-helm --wait --wait-for-jobs --timeout 5m
helm install monitor monitor --namespace neuvector --create-namespace --repo https://neuvector.github.io/neuvector-helm --wait --wait-for-jobs --timeout 5m --values monitor-values.json

monitor-values.json contains:

{"exporter":{"apiSvc":"neuvector-svc-controller:10443"}}

Results in the monitor crashlooping on login:

NAME                                                 READY   STATUS             RESTARTS      AGE
neuvector-cert-upgrader-job-hbl28                    0/1     Completed          0             6m56s
neuvector-controller-pod-647f8d69c5-8jdc5            1/1     Running            0             6m57s
neuvector-controller-pod-647f8d69c5-n652j            1/1     Running            0             6m57s
neuvector-controller-pod-647f8d69c5-qx6hp            1/1     Running            0             6m57s
neuvector-enforcer-pod-pphcm                         1/1     Running            0             6m57s
neuvector-manager-pod-66f865dc7c-vmxgp               1/1     Running            0             6m57s
neuvector-prometheus-exporter-pod-66c569f568-86vn7   0/1     CrashLoopBackOff   6 (52s ago)   6m27s
neuvector-scanner-pod-6b45987549-7ppkq               1/1     Running            0             6m57s
neuvector-scanner-pod-6b45987549-rgcdm               1/1     Running            0             6m57s
neuvector-scanner-pod-6b45987549-zfz85               1/1     Running            0             6m57s

logs from neuvector-prometheus-exporter:

Login to controller ...
Authentication failed

Eventually it turns to:

Login to controller ...
Temporarily blocked because of too many login failures

If I add:

{
  "bootstrapPassword": "admin"
}

As a values file for core and redeploy as above neuvector-prometheus-exporter still crashloops and I get an error:

Login to controller ...
Traceback (most recent call last):
  File "/usr/local/bin/nv_exporter.py", line 645, in <module>
    if _login("https://" + CTRL_SVC, CTRL_USER, CTRL_PASS) < 0:
       ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/nv_exporter.py", line 58, in _login
    token = json.loads(response.text)["token"]["token"]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

neuvector/neuvector#2085 just merged as part of the latest version (https://github.com/neuvector/neuvector/releases/tag/v5.4.6) maybe it's related?

Deploying a test cluster with a default password for testing seemed to work as of a few days ago, so the change seems recent.

[kwmonroe]

I think this is because 5.4.6 now requires you to change the admin password before the REST API will return a valid token, and a valid token is needed by prom-exporter in the monitor chart.

You can work around this deploying the crd and core charts, then change the admin password either via the web ui or api:

curl -k -H \
  "Content-Type: application/json" \
  -d '{"password": {"username": "admin", "password": "'${BOOTSTRAP_PW}'", "new_password": "'${CHANGED_PW}'"}}' \
  "https://${API_SVC_IP}:10443/v1/auth"

Then you can configure the monitor chart with exporter.CTRL_PASSWORD = ${CHANGED_PW}' and deploy that. It would be nice if the core chart exposed a way to handle this initial password reset.