Merge pull request ceph#59544 from VallariAg/wip-teuthology-workflow

doc: update tests-integration-testing-teuthology-workflow.rst

Reviewed-by: Zac Dover <[email protected]>
Reviewed-by: Anthony D'Atri <[email protected]>

Author: zdover23

doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst

@@ -3,11 +3,68 @@
 Integration Tests using Teuthology Workflow
 ===========================================
 
-Scheduling Test Run
--------------------
+Infrastructure
+--------------
+
+Components:
+
+1. `ceph-ci`_: Clone of the main Ceph repository, used for triggering Jenkins 
+   Ceph builds for development.
+
+2. `Ceph Jenkins`_: Responsible for triggering builds, uploading packages 
+   to Chacra, and pushing updates about the build to Shaman.
+
+3. `Shaman`_: UI Interface used to check build status. In its backend, 
+   it is a REST API to query and store build information.
+
+4. `Chacra`_: Service where packages are uploaded. The binaries uploaded 
+   here can be downloaded and used by anyone.
+
+5. `Teuthology CLI`_: Developers can use various Teuthology commands to schedule 
+   and manage test runs.
+
+6. Teuthology: This component is responsible for pushing test jobs to 
+   the Beanstalk queue and Paddles. It also picks jobs from 
+   the queue and runs tests.
+
+7. Beanstalk queue: A priority queue containing all the queued jobs. 
+   Developers typically do not need to interact with it.
+
+8. Paddles: A backend service that stores all test run information. 
+   Developers typically do not need to interact with it.
+
+9. `Pulpito`_: A UI interface (for information stored in Paddles) that allows 
+   developers to see detailed information about their scheduled tests, 
+   including status and results.
+
+10. Testnodes: A cluster of various machines that are used for running tests.
+    Developers usually schedule tests to run on `smithi`_ machines, which are
+    dedicated test nodes for Teuthology integration testing.
+
+Each Teuthology test *run* contains multiple test *jobs*. Each job runs in an 
+environment isolated from other jobs, on a different collection of test nodes.
+
+To test a change in Ceph, follow these steps:
+
+1. Getting binaries - Build Ceph.
+2. Scheduling Test Run:
+
+   a. About Test Suites.
+   b. Triggering Teuthology Tests.
+   c. Testing QA changes (without re-building binaries).
+   d. Filtering Tests.
 
-Getting binaries
-****************
+3. Viewing Test Results:
+
+   a. Pulpito Dashboard.
+   b. Teuthology Archives (Reviewing Logs).
+
+4. Killing tests.
+5. Re-running tests.
+
+
+Getting binaries - Build Ceph
+-----------------------------
 
 Ceph binaries must be built for your branch before you can use teuthology to run integration tests on them. Follow these steps to build the Ceph binaries:
 
@@ -41,8 +98,44 @@ Ceph binaries must be built for your branch before you can use teuthology to run
 .. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph
 
 
-Triggering Tests
-****************
+Naming the ceph-ci branch
+*************************
+Prepend your branch with your name before you push it to ceph-ci. For example,
+a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
+``$yourname`` is replaced with your name. Identifying your branch with your
+name makes your branch easily findable on Shaman and Pulpito.
+
+If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
+the name of that stable branch in your ceph-ci branch name.
+For example, the ``feature-x`` PR branch should be named 
+``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
+
+You can choose to only trigger a CentOS 9.Stream build (excluding other distro like ubuntu)
+by adding "centos9-only" at the end of the ceph-ci branch name. For example,
+``wip-$yourname-feature-centos9-only``. This helps to get quicker builds and save resources 
+when you don't require binaries for other distros. 
+
+Delete the branch from ceph-ci when you no longer need it. If you are
+logged in to GitHub, all your branches on ceph-ci can be found here:
+https://github.com/ceph/ceph-ci/branches.
+
+
+Scheduling Test Run
+-------------------
+
+About Test Suites 
+*****************
+
+Integration tests are organized into “suites”, which are defined in ``qa/suites``
+sub-directory of the Ceph repository. These test suites can be run with the teuthology-suite 
+command. 
+
+See `Suites Inventory`_ for a list of available suites of integration tests.
+
+More details understanding of how these test suites are defined can be found on `Integration Test Introduction Page`_.
+
+Triggering Teuthology Tests
+***************************
 
 After you have built Ceph binaries for your branch, you can run tests using
 teuthology. This procedure explains how to run tests using teuthology.
@@ -54,7 +147,10 @@ teuthology. This procedure explains how to run tests using teuthology.
        ssh <username>@teuthology.front.sepia.ceph.com
 
    This requires Sepia lab access. To request access to the Sepia lab, see:
-   https://ceph.github.io/sepia/adding_users/
+   https://ceph.github.io/sepia/adding_users/.
+
+#. For initial setup, follow `teuthology installation guide`_ to setup teuthology for 
+   your user on teuthology machine. This will enable you to run teuthology commands. 
 
 #. Run the ``teuthology-suite`` command:
 
@@ -66,7 +162,7 @@ teuthology. This procedure explains how to run tests using teuthology.
         -s fs \
         -p 110 \
         --filter "cephfs-shell" \
-        -e [email protected] \
+        -e [email protected]
 
    The options in the above command are defined here: 
 
@@ -101,10 +197,13 @@ teuthology. This procedure explains how to run tests using teuthology.
    `Pulpito`_ where the test results can be viewed.
 
 
+The ``--dry-run`` option allows you to demo-run ``teuthology-suite`` command without 
+actually scheduling teuthology tests. This is helpful to check how many jobs and which jobs
+a command will schedule. 
 
 Other frequently used/useful options are ``-d`` (or ``--distro``),
-``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``,
-``-l`` (for limiting number of jobs) , ``-N`` (for how many times the job will
+``--distro-version``, ``--filter-out``, ``--timeout``, ``flavor``, ``--rerun``,
+``--limit`` (for limiting number of jobs) , ``-N`` (for how many times the job will
 run), and ``--subset`` (used to reduce the number of tests that are triggered). Run
 ``teuthology-suite --help`` to read descriptions of these and other options.
 
@@ -159,15 +258,15 @@ job config printed at the beginning of the teuthology job.
           for the builds to finish, then triggering tests and waiting for 
           the test results. 
 
-About Suites and Filters
-************************
-
-See `Suites Inventory`_ for a list of available suites of integration tests.
-Each directory under ``qa/suites`` in the Ceph repository is an integration
-test suite, and arguments appropriate to follow ``-s`` can be found there.
+Filtering Tests
+***************
 
+Test suites includes combinations of many yaml files which can results in massive 
+amount of jobs being scheduled for a suite. So filters can help to reduce the amount
+of jobs or schedule particular jobs within a suite.
+ 
 Keywords for filtering tests can be found in
-``qa/suites/<suite-name>/<subsuite-name>/tasks`` and can be used as arguments
+``qa/suites/<suite-name>/<subsuite-name>/tasks`` in Ceph repository and can be used as arguments
 for ``--filter``. Each YAML file in that directory can trigger tests; using the
 name of the file without its filename extension as an argument to the
 ``--filter`` triggers those tests. 
@@ -182,6 +281,8 @@ contents of the file for the ``modules`` attribute. For ``cephfs-shell.yaml``
 the ``modules`` attribute is ``tasks.cephfs.test_cephfs_shell``. This means
 that it triggers all tests in ``qa/tasks/cephfs/test_cephfs_shell.py``.
 
+Read more about how to `Filter Tests by their Description`_.
+
 Viewing Test Results
 ---------------------
 
@@ -195,22 +296,26 @@ Teuthology Archives
 *******************
 
 After the tests have finished running, the log for the job can be obtained by
-clicking on the job ID at the Pulpito page associated with your tests. It's
+clicking on the job ID at the Pulpito run page associated with your tests. It's
 more convenient to download the log and then view it rather than viewing it in
-an internet browser since these logs can easily be up to 1 GB in size. It is
-easier to ssh into the teuthology machine (``teuthology.front.sepia.ceph.com``)
-and access the following path::
+an internet browser since these logs can easily be up to 1 GB in size.
+It is also possible to ssh into a `developer playground machine`_ and access the following path::
 
-    /ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
+    /teuthology/<run-name>/<job-id>/teuthology.log
 
 For example: for the above test ID, the path is::
 
-   /ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
+   /teuthology/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
 
 This method can be used to view the log more quickly than would be possible through a browser.
 
-In addition to ``teuthology.log``, some other files are included for debugging
-purposes:
+To view ceph logs (cephadm, ceph monitors, ceph-mgr, etc) or system logs,
+remove ``teuthology.log`` from the job's teuthology log url on browser and then navigate 
+to ``remote/<machine>/log/``. System logs can be found at ``remote/<machine>/syslog/``.
+Similarly, these logs can be found on developer playground machines at 
+``/teuthology/<test-id>/<job-id>/remote/<machine>/``. 
+
+Some other files that are included for debugging purposes:
 
 * ``unit_test_summary.yaml``: Provides a summary of all unit test failures.
   Generated (optionally) when the ``unit_test_scan`` configuration option is
@@ -219,7 +324,7 @@ purposes:
 * ``valgrind.yaml``: Summarizes any Valgrind errors that may occur.
 
 .. note:: To access archives more conveniently, ``/a/`` has been symbolically
-   linked to ``/ceph/teuthology-archive/``. For instance, to access the previous
+   linked to ``/teuthology/``. For instance, to access the previous
    example, we can use something like::
 
    /a/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
@@ -234,9 +339,9 @@ Here is the command that terminates jobs:
 
 .. prompt:: bash $
 
-   teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
+   teuthology-kill -p  -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi -m smithi -o scheduled_teuthology@teuthology 
 
-Let's call the argument passed to ``-r`` as test ID. It can be found
+The argument passed to ``-r`` is run name. It can be found
 easily in the link to the Pulpito page for the tests you triggered. For
 example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
 
@@ -275,28 +380,19 @@ Following's the definition of new options introduced in this section:
                                 'waiting'. Default value: 'fail,dead'
       =======================  ===============================================
 
-Naming the ceph-ci branch
--------------------------
-Prepend your branch with your name before you push it to ceph-ci. For example,
-a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
-``$yourname`` is replaced with your name. Identifying your branch with your
-name makes your branch easily findable on Shaman and Pulpito.
-
-If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
-the name of that stable branch in your ceph-ci branch name.
-For example, the ``feature-x`` PR branch should be named 
-``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
-
-Delete the branch from ceph-ci when you no longer need it. If you are
-logged in to GitHub, all your branches on ceph-ci can be found here:
-https://github.com/ceph/ceph-ci/branches.
-
 .. _ceph-ci: https://github.com/ceph/ceph-ci
+.. _Ceph Jenkins: https://jenkins.ceph.com/
+.. _Teuthology CLI: https://docs.ceph.com/projects/teuthology/en/latest/commands/list.html
 .. _Chacra: https://github.com/ceph/chacra/blob/master/README.rst
 .. _Pulpito: http://pulpito.front.sepia.ceph.com/
 .. _Running Your First Test: ../../running-tests-locally/#running-your-first-test
 .. _Shaman: https://shaman.ceph.com/builds/ceph/
 .. _Suites Inventory: ../tests-integration-testing-teuthology-intro/#suites-inventory
 .. _Testing Priority: ../tests-integration-testing-teuthology-intro/#testing-priority
 .. _Triggering Tests: ../tests-integration-testing-teuthology-workflow/#triggering-tests
+.. _Integration Test Introduction Page: ../tests-integration-testing-teuthology-intro/#how-integration-tests-are-defined
 .. _tests-sentry-developers-guide: ../tests-sentry-developers-guide/
+.. _smithi: https://wiki.sepia.ceph.com/doku.php?id=hardware:smithi
+.. _teuthology installation guide: https://docs.ceph.com/projects/teuthology/en/latest/INSTALL.html#installation-and-setup
+.. _Filter Tests by their Description: ../tests-integration-testing-teuthology-intro/#filtering-tests-by-their-description
+.. _developer playground machine: https://wiki.sepia.ceph.com/doku.php?id=devplayground