RUST-1867 Convert retryable reads legacy tests to unified (#1139)

Author: abr-egn

src/test/spec/json/retryable-reads/README.rst

@@ -9,192 +9,18 @@ Retryable Reads Tests
 Introduction
 ============
 
-The YAML and JSON files in the ``legacy`` and ``unified`` sub-directories are platform-independent tests
-that drivers can use to prove their conformance to the Retryable Reads spec. Tests in the
-``unified`` directory are written using the `Unified Test Format <../../unified-test-format/unified-test-format.rst>`_.
-Tests in the ``legacy`` directory are written using the format described below.
+The YAML and JSON files in this directory are platform-independent tests meant
+to exercise a driver's implementation of retryable reads. These tests utilize
+the [Unified Test Format](../../unified-test-format/unified-test-format.md).
 
-Prose tests, which are not easily expressed in YAML, are also presented
+Several prose tests, which are not easily expressed in YAML, are also presented
 in this file. Those tests will need to be manually implemented by each driver.
 
-Tests will require a MongoClient created with options defined in the tests.
-Integration tests will require a running MongoDB cluster with server versions
-4.0 or later.
+Prose Tests
+===========
 
-N.B. The spec specifies 3.6 as the minimum server version: however,
-``failCommand`` is not supported on 3.6, so for now, testing requires MongoDB
-4.0. Once `DRIVERS-560`_ is resolved, we will attempt to adapt its live failure
-integration tests to test Retryable Reads on MongoDB 3.6.
-
-.. _DRIVERS-560: https://jira.mongodb.org/browse/DRIVERS-560
-
-Server Fail Point
-=================
-
-See: `Server Fail Point`_ in the Transactions spec test suite.
-
-.. _Server Fail Point: ../../transactions/tests#server-fail-point
-
-Disabling Fail Point after Test Execution
------------------------------------------
-
-After each test that configures a fail point, drivers should disable the
-``failCommand`` fail point to avoid spurious failures in
-subsequent tests. The fail point may be disabled like so::
-
-    db.runCommand({
-        configureFailPoint: "failCommand",
-        mode: "off"
-    });
-
-Network Error Tests
-===================
-
-Network error tests are expressed in YAML and should be run against a standalone,
-shard cluster, or single-node replica set.
-
-
-Test Format
------------
-
-Each YAML file has the following keys:
-
-- ``runOn`` (optional): An array of server version and/or topology requirements
-  for which the tests can be run. If the test environment satisfies one or more
-  of these requirements, the tests may be executed; otherwise, this file should
-  be skipped. If this field is omitted, the tests can be assumed to have no
-  particular requirements and should be executed. Each element will have some or
-  all of the following fields:
-
-  - ``minServerVersion`` (optional): The minimum server version (inclusive)
-    required to successfully run the tests. If this field is omitted, it should
-    be assumed that there is no lower bound on the required server version.
-
-  - ``maxServerVersion`` (optional): The maximum server version (inclusive)
-    against which the tests can be run successfully. If this field is omitted,
-    it should be assumed that there is no upper bound on the required server
-    version.
-
-  - ``topology`` (optional): An array of server topologies against which the
-    tests can be run successfully. Valid topologies are "single",
-    "replicaset", "sharded", and "load-balanced". If this field is omitted,
-    the default is all topologies (i.e. ``["single", "replicaset", "sharded",
-    "load-balanced"]``).
-
-  - ``serverless``: (optional): Whether or not the test should be run on Atlas
-    Serverless instances. Valid values are "require", "forbid", and "allow". If
-    "require", the test MUST only be run on Atlas Serverless instances. If
-    "forbid", the test MUST NOT be run on Atlas Serverless instances. If omitted
-    or "allow", this option has no effect.
-
-    The test runner MUST be informed whether or not Atlas Serverless is being
-    used in order to determine if this requirement is met (e.g. through an
-    environment variable or configuration option).
-
-    Note: the Atlas Serverless proxy imitates mongos, so the test runner is not
-    capable of determining if Atlas Serverless is in use by issuing commands
-    such as ``buildInfo`` or ``hello``. Furthermore, connections to Atlas
-    Serverless use a load balancer, so the topology will appear as
-    "load-balanced".
-
-- ``database_name`` and ``collection_name``: Optional. The database and
-  collection to use for testing.
-
-- ``bucket_name``: Optional. The GridFS bucket name to use for testing.
-
-- ``data``: The data that should exist in the collection(s) under test before
-  each test run. This will typically be an array of documents to be inserted
-  into the collection under test (i.e. ``collection_name``); however, this field
-  may also be an object mapping collection names to arrays of documents to be
-  inserted into the specified collection.
-
-- ``tests``: An array of tests that are to be run independently of each other.
-  Each test will have some or all of the following fields:
-
-  - ``description``: The name of the test.
-
-  - ``clientOptions``: Optional, parameters to pass to MongoClient().
-
-  - ``useMultipleMongoses`` (optional): If ``true``, and the topology type is
-    ``Sharded``, the MongoClient for this test should be initialized with multiple
-    mongos seed addresses. If ``false`` or omitted, only a single mongos address
-    should be specified.
-
-    If ``true``, the topology type is ``LoadBalanced``, and Atlas Serverless is
-    not being used, the MongoClient for this test should be initialized with the
-    URI of the load balancer fronting multiple servers. If ``false`` or omitted,
-    the MongoClient for this test should be initialized with the URI of the load
-    balancer fronting a single server.
-
-    ``useMultipleMongoses`` only affects ``Sharded`` and ``LoadBalanced``
-    topologies (excluding Atlas Serverless).
-
-  - ``skipReason``: Optional, string describing why this test should be skipped.
-
-  - ``failPoint``: Optional, a server fail point to enable, expressed as the
-    configureFailPoint command to run on the admin database.
-
-  - ``operations``: An array of documents describing an operation to be
-    executed. Each document has the following fields:
-
-    - ``name``: The name of the operation on ``object``.
-
-    - ``object``: The name of the object to perform the operation on. Can be
-      "database", "collection", "client", or "gridfsbucket."
-
-    - ``arguments``: Optional, the names and values of arguments.
-
-    - ``result``: Optional. The return value from the operation, if any. This
-      field may be a scalar (e.g. in the case of a count), a single document, or
-      an array of documents in the case of a multi-document read.
-
-    - ``error``: Optional. If ``true``, the test should expect an error or
-      exception.
-
-  - ``expectations``: Optional list of command-started events.
-
-GridFS Tests
-------------
-
-GridFS tests are denoted by when the YAML file contains ``bucket_name``.
-The ``data`` field will also be an object, which maps collection names
-(e.g. ``fs.files``) to an array of documents that should be inserted into
-the specified collection.
-
-``fs.files`` and ``fs.chunks`` should be created in the database
-specified by ``database_name``. This could be done via inserts or by
-creating GridFSBuckets—using the GridFS ``bucketName`` (see
-`GridFSBucket spec`_) specified by ``bucket_name`` field in the YAML
-file—and calling ``upload_from_stream_with_id`` with the appropriate
-data.
-
-``Download`` tests should be tested against ``GridFS.download_to_stream``.
-``DownloadByName`` tests should be tested against
-``GridFS.download_to_stream_by_name``.
-
-
-.. _GridFSBucket spec: https://github.com/mongodb/specifications/blob/master/source/gridfs/gridfs-spec.rst#configurable-gridfsbucket-class
-
-
-Speeding Up Tests
------------------
-
-Drivers can greatly reduce the execution time of tests by setting `heartbeatFrequencyMS`_
-and `minHeartbeatFrequencyMS`_ (internally) to a small value (e.g. 5ms), below what
-is normally permitted in the SDAM spec. If a test specifies an explicit value for
-heartbeatFrequencyMS (e.g. client or URI options), drivers MUST use that value.
-
-.. _minHeartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#minheartbeatfrequencyms
-.. _heartbeatFrequencyMS: ../../server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms
-
-Optional Enumeration Commands
-=============================
-
-A driver only needs to test the optional enumeration commands it has chosen to
-implement (e.g. ``Database.listCollectionNames()``).
-
-PoolClearedError Retryability Test
-==================================
+1. PoolClearedError Retryability Test
+-------------------------------------
 
 This test will be used to ensure drivers properly retry after encountering PoolClearedErrors.
 It MUST be implemented by any driver that implements the CMAP specification.
@@ -232,79 +58,96 @@ This test requires MongoDB 4.2.9+ for ``blockConnection`` support in the failpoi
 
 9. Disable the failpoint.
 
-Retrying Reads in a Sharded Cluster
-===================================
+2. Retrying Reads in a Sharded Cluster
+--------------------------------------
 
 These tests will be used to ensure drivers properly retry reads on a different
 mongos.
 
-Retryable Reads Are Retried on a Different mongos if One is Available
----------------------------------------------------------------------
+Note: this test cannot reliably distinguish "retry on a different mongos due to
+server deprioritization" (the behavior intended to be tested) from "retry on a
+different mongos due to normal SDAM behavior of randomized suitable server
+selection". Verify relevant code paths are correctly executed by the tests using
+external means such as a logging, debugger, code coverage tool, etc.
+
+2.1 Retryable Reads Are Retried on a Different mongos When One is Available
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This test MUST be executed against a sharded cluster that has at least two
-mongos instances.
+mongos instances, supports ``retryReads=true``, and has enabled the
+``configureFailPoint`` command (MongoDB 4.2+).
 
-1. Ensure that a test is run against a sharded cluster that has at least two
-   mongoses. If there are more than two mongoses in the cluster, pick two to
-   test against.
+1. Create two clients ``s0`` and ``s1`` that each connect to a single mongos
+   from the sharded cluster. They must not connect to the same mongos.
 
-2. Create a client per mongos using the direct connection, and configure the
-   following fail points on each mongos::
+2. Configure the following fail point for both ``s0`` and ``s1``::
 
      {
          configureFailPoint: "failCommand",
          mode: { times: 1 },
          data: {
              failCommands: ["find"],
-             errorCode: 6,
-             closeConnection: true
+             errorCode: 6
          }
      }
 
-3. Create a client with ``retryReads=true`` that connects to the cluster,
-   providing the two selected mongoses as seeds.
+3. Create a client ``client`` with ``retryReads=true`` that connects to the
+   cluster using the same two mongoses as ``s0`` and ``s1``.
+
+4. Enable failed command event monitoring for ``client``.
+
+5. Execute a ``find`` command with ``client``. Assert that the command failed.
 
-4. Enable command monitoring, and execute a ``find`` command that is
-   supposed to fail on both mongoses.
+6. Assert that two failed command events occurred. Assert that both events
+   occurred on different mongoses.
 
-5. Asserts that there were failed command events from each mongos.
+7. Disable the fail point on both ``s0`` and ``s1``.
 
-6. Disable the fail points.
 
+2.2 Retryable Reads Are Retried on the Same mongos When No Others are Available
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Retryable Reads Are Retried on the Same mongos if No Others are Available
--------------------------------------------------------------------------
+This test MUST be executed against a sharded cluster that supports
+``retryReads=true`` and has enabled the ``configureFailPoint`` command 
+(MongoDB 4.2+).
 
-1. Ensure that a test is run against a sharded cluster. If there are multiple
-   mongoses in the cluster, pick one to test against.
+1. Create a client ``s0`` that connects to a single mongos from the cluster.
 
-2. Create a client that connects to the mongos using the direct connection,
-   and configure the following fail point on the mongos::
+2. Configure the following fail point for ``s0``::
 
      {
          configureFailPoint: "failCommand",
          mode: { times: 1 },
          data: {
              failCommands: ["find"],
-             errorCode: 6,
-             closeConnection: true
+             errorCode: 6
          }
      }
 
-3. Create a client with ``retryReads=true`` that connects to the cluster,
-   providing the selected mongos as the seed.
+3. Create a client ``client`` with ``directConnection=false`` (when not set by
+   default) and ``retryReads=true`` that connects to the cluster using the same
+   single mongos as ``s0``.
 
-4. Enable command monitoring, and execute a ``find`` command.
+4. Enable succeeded and failed command event monitoring for ``client``.
 
-5. Asserts that there was a failed command and a successful command event.
+5. Execute a ``find`` command with ``client``. Assert that the command
+   succeeded.
 
-6. Disable the fail point.
+6. Assert that exactly one failed command event and one succeeded command event
+   occurred. Assert that both events occurred on the same mongos.
+
+7. Disable the fail point on ``s0``.
 
 
 Changelog
 =========
 
-:2023-08-26 Add prose tests for retrying in a sharded cluster.
+:2024-03-06: Convert legacy retryable reads tests to unified format.
+
+:2024-02-21: Update mongos redirection prose tests to workaround SDAM behavior
+             preventing execution of deprioritization code paths.
+
+:2023-08-26: Add prose tests for retrying in a sharded cluster.
 
 :2022-04-22: Clarifications to ``serverless`` and ``useMultipleMongoses``.