Only enable CI mode is $CI or $BUILD_NUMBER is set to a non-empty string

Previously, if either env variable was defined, CI mode would be activated,
even if the variable was set to the empty string.
This had some unfortunate consequences:
* The easy and obvious method of setting those variables,
  i.e. `CI="${OTHER_VAR}"`,
  would not work,
  because `$CI` would always end up being defined
  even if `$OTHER_VAR` was undefined.
* The easy and obvious method of clearing those variables at runtime,
  i.e. `CI="" pytest ...`,
  would not work,
  because again `$CI` would still be defined even though it was blank.

Now, at least one of those variables must be set to a non-empty string
in order to enabled CI mode.

Closes #13766

Author: Jordan Macdonald

AUTHORS

@@ -227,6 +227,7 @@ Jon Parise
 Jon Sonesen
 Jonas Obrist
 Jordan Guymon
+Jordan Macdonald
 Jordan Moldow
 Jordan Speicher
 Joseph Hunkeler

changelog/13766.breaking.rst

@@ -0,0 +1,2 @@
+Previously, pytest would assume it was running in a CI/CD environment if either of the environment variables `$CI` or `$BUILD_NUMBER` was defined;
+now, CI mode is only activated if at least one of those variables is defined and set to a *non-empty* value.

doc/en/explanation/ci.rst

@@ -17,8 +17,7 @@ adapt some of its behaviours.
 How CI is detected
 ------------------
 
-Pytest knows it is in a CI environment when either one of these environment variables are set,
-regardless of their value:
+Pytest knows it is in a CI environment when either one of these environment variables are set to a non-empty value:
 
 * `CI`: used by many CI systems.
 * `BUILD_NUMBER`: used by Jenkins.

doc/en/reference/reference.rst

@@ -1165,11 +1165,11 @@ Environment variables that can be used to change pytest's behavior.
 
 .. envvar:: CI
 
-When set (regardless of value), pytest acknowledges that is running in a CI process. Alternative to ``BUILD_NUMBER`` variable. See also :ref:`ci-pipelines`.
+When set to a non-empty value, pytest acknowledges that is running in a CI process. See also :ref:`ci-pipelines`.
 
 .. envvar:: BUILD_NUMBER
 
-When set (regardless of value), pytest acknowledges that is running in a CI process. Alternative to CI variable. See also :ref:`ci-pipelines`.
+When set to a non-empty value, pytest acknowledges that is running in a CI process. Alternative to :envvar:`CI`. See also :ref:`ci-pipelines`.
 
 .. envvar:: PYTEST_ADDOPTS
 
@@ -2408,7 +2408,7 @@ All the command-line flags can be obtained by running ``pytest --help``::
                             Plugins that must be present for pytest to run
 
     Environment variables:
-      CI                       When set (regardless of value), pytest knows it is running in a CI process and does not truncate summary info
+      CI                       When set to a non-empty value, pytest knows it is running in a CI process and does not truncate summary info
       BUILD_NUMBER             Equivalent to CI
       PYTEST_ADDOPTS           Extra command line options
       PYTEST_PLUGINS           Comma-separated plugins to load during startup

src/_pytest/compat.py

@@ -308,5 +308,6 @@ def __call__(self) -> bool:
 
 def running_on_ci() -> bool:
     """Check if we're currently running on a CI system."""
+    # Only enable CI mode if one of these env variables is defined and non-empty.
     env_vars = ["CI", "BUILD_NUMBER"]
-    return any(var in os.environ for var in env_vars)
+    return any(os.environ.get(var) for var in env_vars)

src/_pytest/helpconfig.py

@@ -221,7 +221,7 @@ def showhelp(config: Config) -> None:
     vars = [
         (
             "CI",
-            "When set (regardless of value), pytest knows it is running in a "
+            "When set to a non-empty value, pytest knows it is running in a "
             "CI process and does not truncate summary info",
         ),
         ("BUILD_NUMBER", "Equivalent to CI"),

testing/test_assertion.py

@@ -567,6 +567,11 @@ def test_full_diff():
         result = pytester.runpytest()
         result.stdout.fnmatch_lines(["E         Full diff:"])
 
+        # Setting CI to empty string is same as having it undefined
+        monkeypatch.setenv("CI", "")
+        result = pytester.runpytest()
+        result.stdout.fnmatch_lines(["E         Use -v to get more diff"])
+
         monkeypatch.delenv("CI", raising=False)
         result = pytester.runpytest()
         result.stdout.fnmatch_lines(["E         Use -v to get more diff"])
@@ -1465,6 +1470,17 @@ def test_many_lines():
         result = pytester.runpytest("-vv")
         result.stdout.fnmatch_lines(["* 6*"])
 
+        # Setting CI to empty string is same as having it undefined
+        monkeypatch.setenv("CI", "")
+        result = pytester.runpytest()
+        result.stdout.fnmatch_lines(
+            [
+                "*+ 1*",
+                "*+ 3*",
+                f"*truncated ({expected_truncated_lines} lines hidden)*use*-vv*",
+            ]
+        )
+
         monkeypatch.setenv("CI", "1")
         result = pytester.runpytest()
         result.stdout.fnmatch_lines(["* 6*"])

testing/test_faulthandler.py

@@ -79,7 +79,7 @@ def test_disabled():
         pytest.param(
             True,
             marks=pytest.mark.skipif(
-                "CI" in os.environ
+                bool(os.environ.get("CI"))
                 and sys.platform == "linux"
                 and sys.version_info >= (3, 14),
                 reason="sometimes crashes on CI because of truncated outputs (#7022)",