Flaky auth failure on cluster setup

[ShaneHarvey]

Flaky auth failure on cluster setup:

https://spruce.mongodb.com/task/mongo_python_driver_tests_python_version_rhel8_test_ssl__platform~rhel8_auth_ssl~auth_ssl_python_version~3.7_coverage~coverage_test_4.2_replica_set_8422edf3abced2b9492123bf72ac91ab3586a69d_23_12_07_21_46_46/logs?execution=0

 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37373, host=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.servers:342 - pid=37373, hostname=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37379, host=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.servers:342 - pid=37379, hostname=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,869 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37383, host=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,870 [DEBUG] mongo_orchestration.servers:342 - pid=37383, hostname=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,881 [ERROR] mongo_orchestration.apps:68 - <function rs_create at 0x7fef4a9a3920>
 [2023/12/07 14:07:00.236] Traceback (most recent call last):
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/__init__.py", line 66, in wrap
 [2023/12/07 14:07:00.236]     return f(*arg, **kwd)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 80, in rs_create
 [2023/12/07 14:07:00.236]     result = _rs_create(data)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 37, in _rs_create
 [2023/12/07 14:07:00.236]     rs_id = ReplicaSets().create(params)
 [2023/12/07 14:07:00.236]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 648, in create
 [2023/12/07 14:07:00.236]     repl = ReplicaSet(rs_params)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 120, in __init__
 [2023/12/07 14:07:00.236]     self.restart_with_auth()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 159, in restart_with_auth
 [2023/12/07 14:07:00.236]     self.restart(config_callback=add_auth)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 617, in restart
 [2023/12/07 14:07:00.236]     f.result()
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
 [2023/12/07 14:07:00.236]     return self.__get_result()
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
 [2023/12/07 14:07:00.236]     raise self._exception
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
 [2023/12/07 14:07:00.236]     result = self.fn(*self.args, **self.kwargs)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 449, in restart
 [2023/12/07 14:07:00.236]     return self.start(timeout)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 351, in start
 [2023/12/07 14:07:00.236]     self.run_command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 272, in run_command
 [2023/12/07 14:07:00.236]     result = getattr(self.connection.admin, mode)(command, name, **d)
 [2023/12/07 14:07:00.236]                      ^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 215, in connection
 [2023/12/07 14:07:00.236]     connected(c)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/common.py", line 130, in connected
 [2023/12/07 14:07:00.236]     client.admin.command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
 [2023/12/07 14:07:00.236]     return func(self, *args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/database.py", line 890, in command
 [2023/12/07 14:07:00.236]     with self.__client._conn_for_reads(read_preference, session) as (
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1330, in _conn_from_server
 [2023/12/07 14:07:00.236]     with self._checkout(server, session) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1252, in _checkout
 [2023/12/07 14:07:00.236]     with server.checkout(handler=err_handler) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1637, in checkout
 [2023/12/07 14:07:00.236]     conn = self._get_conn(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1756, in _get_conn
 [2023/12/07 14:07:00.236]     conn = self.connect(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1607, in connect
 [2023/12/07 14:07:00.236]     conn.authenticate()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1078, in authenticate
 [2023/12/07 14:07:00.236]     auth.authenticate(creds, self, reauthenticate=reauthenticate)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 625, in authenticate
 [2023/12/07 14:07:00.236]     auth_func(credentials, conn)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 530, in _authenticate_default
 [2023/12/07 14:07:00.236]     return _authenticate_scram(credentials, conn, "SCRAM-SHA-1")
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 257, in _authenticate_scram
 [2023/12/07 14:07:00.236]     res = conn.command(source, cmd)
 [2023/12/07 14:07:00.236]           ^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 322, in inner
 [2023/12/07 14:07:00.236]     return func(*args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 968, in command
 [2023/12/07 14:07:00.236]     return command(
 [2023/12/07 14:07:00.236]            ^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/network.py", line 192, in command
 [2023/12/07 14:07:00.236]     helpers._check_command_response(
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 230, in _check_command_response
 [2023/12/07 14:07:00.236]     raise OperationFailure(errmsg, code, response, max_wire_version)
 [2023/12/07 14:07:00.236] pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,886 [DEBUG] mongo_orchestration.apps:55 - send_result(500)
 [2023/12/07 14:07:00.240] curl: (22) The requested URL returned error: 500 Internal Server Error

Perhaps the user was rolled back in the restart process? We should double check that we are using write concern majority when adding users.

[ShaneHarvey]

We already create the user with the correct write concern:

        # Explicitly set write concern to number of data-bearing members.
        # If we add a user later, we need to guarantee that every node
        # has the user before we authenticate ('majority' is insufficient).
        self._write_concern = len(
            [m for m in members
             if not m.get('rsParams', {}).get('arbiterOnly')]
        )

[ShaneHarvey]

This error can happen when an existing cluster is running on port X and we try to start a new cluster on the same port.