Merge branch 'testing-docs'

Author: jsl12

docs/INTERNALS.rst

@@ -56,7 +56,7 @@ to run in the correct order, which is the reverse order that the contexts were e
 Contexts
 ~~~~~~~~
 
-The various context managers that get entered as AppDaemon is started include the logic for these steps. Some of these
+The various context managers that get entered as AppDaemon is started include the logic for following steps. Some of these
 are entered as part of the ``ADMain`` context, and some are entered in the :py:class:`~appdaemon.__main__.ADMain.run`
 method. All of them are exited in reverse order as the :py:class:`~contextlib.ExitStack` is closed, which happens when
 ``ADMain`` context is exited.

docs/TESTING.rst

@@ -0,0 +1,93 @@
+Testing AppDaemon
+=================
+
+AppDaemon uses `pytest <https://docs.pytest.org/en/stable/>`_ and `pytest-asyncio <https://pytest-asyncio.readthedocs.io/en/stable/>`_.
+
+Background
+----------
+
+- Pytest is configured in the this section in the `pyproject.toml <https://docs.pytest.org/en/stable/reference/customize.html#pyproject-toml>`_ file.
+
+  .. literalinclude:: ../pyproject.toml
+     :language: toml
+     :lines: 93-102
+     :caption: Pytest configuration options
+
+- Pytest-asyncio manages creating the event loop, which is normally handled by :py:class:`~appdaemon.__main__.ADMain`.
+  The event loop persists throughout the test session and is reused many times by different instantiations of the
+  top-level :py:class:`~appdaemon.appdaemon.AppDaemon` class.
+- Instantiating the :py:class:`~appdaemon.appdaemon.AppDaemon` class can now be done without any side effects. This will
+  also instantiate the necessary subsystem classes, but nothing will really start happening until the :py:meth:`~appdaemon.appdaemon.AppDaemon.start` method is called.
+- The :py:class:`~appdaemon.appdaemon.AppDaemon` object is provided as a `pytest fixture <https://docs.pytest.org/en/stable/how-to/fixtures.html#>`_ (``ad``)
+  with the ``function`` scope, which means it will be recreated fresh for each test function. The fixture handles starting
+  and stopping the :py:class:`~appdaemon.appdaemon.AppDaemon` instance before/after each test. It also disables all apps,
+  so they can be selectively run by the tests.
+- Apps can be modified before they're run by the :py:class:`~appdaemon.app_management.AppManagement` object.
+- The ``run_app_for_time`` fixture combines the functionality of the ``ad`` fixture with the
+  :py:meth:`~appdaemon.app_management.AppManagement.app_run_context` so that apps can be temporarily modified and run
+  for short periods.
+
+
+Running Tests
+-------------
+
+Use the `uv run <https://docs.astral.sh/uv/reference/cli/#uv-run>`_ command to ensure uv handles the environment. It will make sure the dependencies are all satisfied.
+
+.. code-block:: bash
+   :caption: Run all tests
+
+   uv run pytest
+
+
+Unit Tests
+~~~~~~~~~~
+
+Unit tests don't require AppDaemon to be running and should be run frequently during development. Currently the only unit tests are ones that cover datetime and timedelta parsing.
+
+.. code-block:: bash
+   :caption: Run unit tests
+
+   uv run pytest -m unit
+
+
+Functional
+~~~~~~~~~~
+
+Functional tests cover various end-to-end interactions between components, so they require AppDaemon to be running. An example would be starting an app, having it register a callback for an event, firing that event, and checking that the callback was called. These should cover as many corner cases as possible
+
+.. code-block:: bash
+   :caption: Run functional tests
+
+   uv run pytest -m functional
+
+Startup Tests
+^^^^^^^^^^^^^
+
+.. autofunction:: tests.functional.test_startup.test_hello_world
+
+Event Tests
+^^^^^^^^^^^
+
+.. autoclass:: tests.functional.test_event.TestEventCallback
+   :members:
+
+State Tests
+^^^^^^^^^^^
+
+.. autoclass:: tests.functional.test_state.TestStateCallback
+   :members:
+
+CI Tests
+~~~~~~~~
+
+The CI tests get run as part of the GitHub action on PRs to the ``dev`` branch. They're intended to each run more or less instantly and collectively only take a few seconds.
+
+.. code-block:: bash
+   :caption: Run CI tests
+
+   uv run pytest -m ci
+
+Plugin Tests
+~~~~~~~~~~~~
+
+Testing plugins involves either connecting to or mocking external systems, so aren't yet covered.

docs/index.rst

@@ -74,6 +74,7 @@ Contents:
    WIDGETDEV
    DEV
    INTERNALS
+   TESTING
    REST_STREAM_API
    UPGRADE_FROM_3.x
    UPGRADE_FROM_2.x

pyproject.toml

@@ -89,15 +89,14 @@ exclude = ["contrib", "docs", "docs.*", "tests*"]
 [tool.setuptools.dynamic]
 version = {attr = "appdaemon.version.__version__"}
 
-# Taken from: https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html
+# https://docs.pytest.org/en/stable/explanation/goodpractices.html
 [tool.pytest.ini_options]
 asyncio_mode = "strict"
 asyncio_default_test_loop_scope = "session"
 asyncio_default_fixture_loop_scope = "session"
-addopts = [
-    "--import-mode=importlib",
-]
+addopts = ["--import-mode=importlib"]
 markers = [
+    "unit: mark test as unit test",
     "ci: mark test to run in CI environment",
     "functional: mark test as functional test",
 ]
@@ -148,7 +147,8 @@ lint.select = ["E", "F"]
 lint.ignore = []
 
 # Allow autofix for all enabled rules (when `--fix`) is provided.
-lint.fixable = ["E", "F", "UP"]
+# lint.fixable = ["E", "F", "UP"]
+lint.fixable = ["ALL"]
 lint.unfixable = []
 
 # Allow unused variables when underscore-prefixed.

tests/conf/apps/event_test_app.py

@@ -18,10 +18,10 @@ class TestEventCallback(ADAPI):
         fire_kwargs: The keyword arguments to pass to the fire_event method
     """
     def initialize(self):
-        self.log(f"{self.__class__.__name__} initialized")
+        self.execute_event = asyncio.Event()
         self.listen_event(self.event_callback, self.event, **self.listen_kwargs)
         self.run_in(self.test_fire, delay=self.args.get("delay", 0.1), **self.fire_kwargs)
-        self.execute_event = asyncio.Event()
+        self.log(f"{self.__class__.__name__} initialized")
 
     @property
     def event(self) -> str:

tests/conf/apps/state_test_app.py

@@ -1,5 +1,6 @@
+import asyncio
 from enum import Enum, auto
-from functools import partial
+from typing import Any
 
 from appdaemon.adapi import ADAPI
 
@@ -27,56 +28,33 @@ class StateTestApp(ADAPI):
     """A simple AppDaemon app to test state management."""
 
     def initialize(self):
+        self.set_log_level("DEBUG")
         self.log("Hello from AppDaemon")
-        self.add_namespace("test", persist=False)
+        self.execute_event = asyncio.Event()
         self.set_namespace("test")
-
         self.add_entity(TEST_ENTITY, state="initialized")
-
-        listen = partial(self.listen_state, self.state_callback, TEST_ENTITY)
-        run_soon = partial(self.run_in, self.change_state, delay=self.delay)
-
-        self.log(f"Running in {self.mode} mode")
-        match self.mode:
-            case StateTestAppMode.ATTRIBUTES:
-                run_soon = partial(run_soon, test_kwarg=self.args["test_kwarg"])
-            case StateTestAppMode.LISTEN_KWARGS:
-                listen = partial(listen, listen_kwarg=self.args["test_kwarg"])
-            case StateTestAppMode.NEW_STATE_FILTER_POSITIVE | StateTestAppMode.NEW_STATE_FILTER_NEGATIVE:
-                listen = partial(listen, new=self.args["new"])
-            case StateTestAppMode.NEW_ATTRIBUTE_FILTER_POSITIVE | StateTestAppMode.NEW_ATTRIBUTE_FILTER_NEGATIVE:
-                listen = partial(listen, attribute=self.args["attribute"], new="changed")
-                attrs = {self.args["attribute"]: self.args["value"]}
-                run_soon = partial(run_soon, **attrs)
-
-        self.log(f"Calling listen_state with kwargs: {listen.keywords}")
-        listen()
-
-        change_state_kwargs = run_soon.keywords.copy()
-        change_state_kwargs.pop("delay", None)
-        self.log(f"Calling {run_soon.args[0].__name__} with kwargs: {change_state_kwargs}")
-        run_soon()
+        self.listen_state(self.state_callback, TEST_ENTITY, **self.listen_kwargs)
+        self.run_in(self.test_change_state, delay=self.delay, **self.state_kwargs)
 
     @property
     def delay(self) -> float:
         return self.args.get("delay", 0.1)
 
     @property
-    def mode(self) -> StateTestAppMode:
-        return StateTestAppMode(self.args.get("mode", StateTestAppMode.BASIC))
-
-    def change_state(self, **kwargs) -> None:
-        """Change the state of the test_state entity."""
-        kwargs.pop("__thread_id", None)
-        match self.mode:
-            case StateTestAppMode.BASIC:
-                self.set_state(TEST_ENTITY, state="changed")
-            case StateTestAppMode.NEW_ATTRIBUTE_FILTER_POSITIVE | StateTestAppMode.NEW_ATTRIBUTE_FILTER_NEGATIVE:
-                self.set_state(TEST_ENTITY, attributes=kwargs)
-            case _:
-                self.set_state(TEST_ENTITY, state="changed", attributes=kwargs)
-
-    def state_callback(self, entity: str, attribute: str, old: str, new: str, **kwargs) -> None:
+    def listen_kwargs(self) -> dict[str, Any]:
+        return self.args.get("listen_kwargs", {})
+
+    @property
+    def state_kwargs(self) -> dict[str, Any]:
+        return self.args.get("state_kwargs", {})
+
+    def test_change_state(self, **kwargs):
+        self.set_state(TEST_ENTITY, **kwargs)
+
+    def state_callback(self, entity: str, attribute: str, old: Any, new: Any, **kwargs: Any) -> None:
+        assert isinstance(entity, str), "Entity should be a string"
+        assert isinstance(attribute, str), "Attribute should be a string"
+
         self.log(f' {entity}.{attribute} '.center(40, '-'))
         self.log(f"{entity}.{attribute} changed from {old} to {new} with kwargs: {kwargs}")
 
@@ -85,3 +63,4 @@ def state_callback(self, entity: str, attribute: str, old: str, new: str, **kwar
 
         self.log(f"New state for {entity}: {new_state}")
         self.log("State callback executed successfully")
+        self.execute_event.set()

tests/functional/test_event.py

@@ -1,21 +1,18 @@
 import asyncio
 import logging
 import uuid
-from collections.abc import Callable
-from contextlib import AbstractAsyncContextManager
 
 import pytest
 from appdaemon.app_management import ManagedObject
-from appdaemon.appdaemon import AppDaemon
 
-logger = logging.getLogger("AppDaemon._test")
-
-AsyncTempTest = Callable[..., AbstractAsyncContextManager[tuple[AppDaemon, pytest.LogCaptureFixture]]]
+from .utils import AsyncTempTest
 
+logger = logging.getLogger("AppDaemon._test")
 
 @pytest.mark.ci
 @pytest.mark.functional
 class TestEventCallback:
+    """Class to group the various tests for event callbacks."""
     app_name: str = "test_event_app"
 
     @pytest.mark.asyncio(loop_scope="session")
@@ -25,17 +22,17 @@ async def test_event_callback(self, run_app_for_time: AsyncTempTest) -> None:
         Process:
             - Unique values are generated for the event and its kwargs
             - The run_app_for_time context manager is used to run the test_event_app temporarily
-                - Event test app listens for an event and fires the same event shortly after
-            - Wait for the async execute_event to be set by the event callback in the app
-            - Clear the event
+            - Event test app listens for an event and fires the same event shortly after
+            - Wait for :py:class:`~asyncio.Event` to be set by the callback in the app
+            - Clear the :py:class:`~asyncio.Event`
             - Set the app arg to another event name so it will no longer match the event that was initially listened for
             - Fire the new event
-            - Wait for the async execute_event again, expecting a timeout.
+            - Wait for the :py:class:`~asyncio.Event` again, expecting a timeout.
 
         Coverage:
-            - listen_event results in the callback being called for corresponding events
-            - keyword arguments provided in listen_event are passed to the callback in ``kwargs``
-            - keyword arguments provided in the fire_event call are passed to the callback in ``data``
+            - ``listen_event`` results in the callback being called for corresponding events
+            - keyword arguments provided in ``listen_event`` are passed to the callback in ``kwargs``
+            - keyword arguments provided in the ``fire_event`` call are passed to the callback in ``data``
         """
         listen_id = str(uuid.uuid4())
         fire_id = str(uuid.uuid4())
@@ -73,31 +70,32 @@ async def test_event_callback(self, run_app_for_time: AsyncTempTest) -> None:
     async def test_event_callback_filtered(self, run_app_for_time: AsyncTempTest, sign: bool) -> None:
         """Test the event callback filtering based on keyword arguments.
 
-        If the event data has a key that matches one of the kwargs provided in the listen_event call, then the values
+        If the event data has a key that matches one of the kwargs provided in the ``listen_event`` call, then the values
         for those keys must also match for the callback to be executed.
 
+        Process:
+            - A unique value is generated for firing the event. If the callback is supposed to fire (positive case),
+              then the same value is used for listening to the event. Otherwise (negative case), a different, unique
+              value is used to listen for the event, which will prevent the callback from executing.
+            - The unique fire and listen values are passed to the app as args.
+            - The ``test_event_app`` app is run until a python :py:class:`~asyncio.Event` is set.
+            - The :py:class:`~asyncio.Event` is created when the app initializes.
+            - The app listens for the event and then fires it after a short delay, using the relevant kwargs for each.
+            - If the callback is executed, :py:class:`~asyncio.Event` is set, and the unique values are printed in the logs.
+
         Coverage:
-            - Event filtering based on kwargs provided to listen_event
-                - Positive case: There is a matching key between the listen kwargs and the event data and the values
-                    match
-                - Negative case: There is a matching key between the listen kwargs and the event data but the values
-                    do not match
+            - Positive
+                There is a matching key between the listen kwargs and the event data and the values match, so the callback is executed.
+            - Negative
+                There is a matching key between the listen kwargs and the event data but the values do not match, so the callback is not executed.
         """
-        listen_id = str(uuid.uuid4())
         fire_id = str(uuid.uuid4())
+        listen_id = fire_id if sign else str(uuid.uuid4())
         test_kwarg_name = "test_kwarg"
-        if sign:
-            # The kwarg name and value must match for the callback to work.
-            app_args = {
-                "listen_kwargs": {test_kwarg_name: listen_id},
-                "fire_kwargs": {test_kwarg_name: listen_id},
-            }
-        else:
-            # The kwarg name must match, but the value must be different for the callback to be filtered.
-            app_args = {
-                "listen_kwargs": {test_kwarg_name: listen_id},
-                "fire_kwargs": {test_kwarg_name: fire_id},
-            }
+        app_args = {
+            "listen_kwargs": {test_kwarg_name: listen_id},
+            "fire_kwargs": {test_kwarg_name: fire_id},
+        }
 
         async with run_app_for_time(self.app_name, **app_args) as (ad, caplog):
             match ad.app_management.objects.get(self.app_name):
@@ -122,10 +120,15 @@ async def test_event_callback_namespace(self, run_app_for_time: AsyncTempTest, s
 
         Event callbacks should only be fired for events in the correct namespace.
 
+        Process:
+            - The event test app is given namespaces to listen and fire the event in.
+            - The app listens for the event and then fires it after a short delay, using the relevant namespaces for each.
+
         Coverage:
-            - Event callbacks should only be fired for events in the correct namespace.
-                - Positive case: The listen and fire namespaces match
-                - Negative case: The listen and fire namespaces do not match
+            - Positive
+                The listen and fire namespaces match, so the callback is executed.
+            - Negative
+                The listen and fire namespaces do not match, so the callback is not executed.
         """
         namespace = "test"
         if sign:
@@ -160,6 +163,10 @@ async def test_event_callback_oneshot(self, run_app_for_time: AsyncTempTest) ->
 
         Event callbacks that are registered with the oneshot flag should only be fired once.
 
+        Process:
+            - Listen for an event with the oneshot flag set.
+            - Fire the event twice, and ensure that the callback is only fired once.
+
         Coverage:
             - Event callbacks that are registered with the oneshot flag should only be fired once.
         """

tests/functional/test_startup.py

@@ -11,6 +11,8 @@
 @pytest.mark.parametrize("app_name", ["hello_world", "another_app"])
 @pytest.mark.asyncio(loop_scope="session")
 async def test_hello_world(ad: AppDaemon, caplog: pytest.LogCaptureFixture, app_name: str) -> None:
+    """Run one of the hello world apps and ensure that the startup text is in the logs."""
+
     ad.app_dir = ad.config_dir / "apps/hello_world"
     assert ad.app_dir.exists(), "App directory does not exist"
     logger.info("Test started")