Added tutorial part for projections, as part 4, moving tutorial part for systems to part 5.

Also improved project module doc significantly to be more of a reference and less of a tutorial.

Also slightly improved other docs.

Author: johnbywater

docs/topics/persistence.rst

@@ -735,9 +735,10 @@ catastrophic inconsistencies in the state of a system.
 Tracking
 ========
 
-A tracking object identifies the position of an event in an application sequence
-when the consequences of processing that event are to be stored by an event process
-component.
+A tracking object identifies the position of an event in an application sequence.
+This is useful when the results of processing an event are to be stored by
+an event processing component, so that we can ensure each event is processed
+only once, and so that we can resume processing events from the correct position.
 
 The library's :class:`~eventsourcing.persistence.Tracking` class
 is a Python frozen data class.

docs/topics/projection.rst

@@ -2,13 +2,13 @@
 :mod:`~eventsourcing.projection` --- Projections
 ================================================
 
-This module shows how :doc:`event-sourced applications
-</topics/application>` can be projected into materialised
+This module may help you develop event-processing components that project
+the state of an :doc:`event-sourced applications </topics/application>` into materialised
 views that support arbitrary queries.
 
-The central idea here follows the notion from `CQRS <https://en.wikipedia.org/wiki/Command_Query_Responsibility_Segregation>`_
+The central idea of this module follows the notion from `CQRS <https://en.wikipedia.org/wiki/Command_Query_Responsibility_Segregation>`_
 of having separate command and query interfaces. This idea is often implemented in event-sourced systems
-with distinct and separate "write" and "read" models. The "write model" is an event-sourced application,
+by developing distinct and separate "write" and "read" models. The "write model" is an event-sourced application,
 and the "read model" is one or many "materialised views" of the event-sourced application. The event-sourced
 application is projected into a materialised view, by processing the application's events,
 usually with an asynchronous event-processing component, so that the materialised view is
@@ -21,117 +21,162 @@ the tracking records to be unique, and by resuming to process the application fr
 position indicated by the last tracking record, the materialised
 view will be a "reliable" deterministic function of the state of the application.
 
+.. _Subscriptions:
 
-Tracking recorders
-==================
+Application subscriptions
+=========================
 
-The library's :ref:`tracking recorder <Tracking recorder>` classes
-(:class:`~eventsourcing.popo.POPOTrackingRecorder`, :class:`~eventsourcing.sqlite.SQLiteTrackingRecorder`,
-and :class:`~eventsourcing.postgres.PostgresTrackingRecorder`) can be extended arbitrarily to define command
-and query methods that update and present a materialised view of the state of an event-sourced application
-atomically with tracking information. The tracking information indicates the position in an application
-sequence of an event that was processed when the materialised view was updated.
+This module provides an :class:`~eventsourcing.projection.ApplicationSubscription` class, which can
+be used to "subscribe" to the domain events of an application.
 
-For example, the :class:`CountRecorder` class, included below, extends the library's abstract base class
-:class:`~eventsourcing.persistence.TrackingRecorder` by defining abstract methods
-:func:`incr_created_events_counter`, :func:`incr_subsequent_events_counter`, :func:`get_created_events_counter`,
-:func:`get_subsequent_events_counter`, and :func:`get_all_events_counter`. These methods
-will be implemented by concrete tracking recorder classes to update a materialised view
-that counts aggregate events from an event-sourced application.
+Application subscriptions are useful when running an event processing component
+that projects the state of an event-sourced application into a materialised view
+of the state of the application.
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: CountRecorder
+Application subscription objects are iterators that yield all domain events recorded
+in an application sequence. Iterating over an application subscription will block when
+all recorded domain events have been yielded, and then continue when new events are recorded.
 
-The :class:`POPOCountRecorder` class, included below, implements this interface using plain old Python objects.
+Application subscription objects can be constructed with an application object, and an integer
+position in its application sequence (a notification ID). The application subscription will yield
+domain events that have notification IDs greater than the given position.
+
+Application subscription objects use the :func:`~eventsourcing.persistence.ApplicationRecorder.subscribe`
+method of the application's recorder to listen to the application's database, selecting notification objects
+and converting them into domain events using the application's mapper.
+
+Each yielded domain event is accompanied by a tracking object that identifies the position of the
+domain event in its application sequence. The tracking objects yielded by the application subscription
+can be recorded atomically along with the new state that results from processing the domain event.
+
+.. code-block:: python
+
+    from eventsourcing.application import Application
+    from eventsourcing.domain import Aggregate
+    from eventsourcing.projection import ApplicationSubscription
+
+    # Construct an application object.
+    app = Application()
+
+    # Record an event.
+    aggregate = Aggregate()
+    app.save(aggregate)
+
+    # Position in application sequence from which to subscribe.
+    max_tracking_id = 0
+
+    with ApplicationSubscription(app, gt=max_tracking_id) as subscription:
+        for domain_event, tracking in subscription:
+            # Process the event and record new state with tracking information.
+            break  # ...so we can continue with the examples
+
+If an event-processing component is using a :ref:`tracking recorder <Tracking recorder>` to record new state atomically
+with tracking objects, subscriptions can be started from the notification ID returned from the tracking recorder's
+:func:`~eventsourcing.persistence.TrackingRecorder.max_tracking_id` method.
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: POPOCountRecorder
 
 .. _Projection:
 
 Projection
 ==========
 
-The library's abstract base class :class:`~eventsourcing.projection.Projection` can be used to define how
-domain events will be processed. Subclasses are expected to use a particular kind of tracking recorder. It
-defines an abstract method :func:`~eventsourcing.projection.Projection.process_event` that must be implemented by subclasses.
+This module provides a generic abstract base class, :class:`~eventsourcing.projection.Projection`,
+which can be used to define how the domain events of an application will be processed.
 
-For example, the :class:`CountProjection` class, included below, inherits :class:`~eventsourcing.projection.Projection`,
-specifies that instances will use a :class:`CountRecorder`, and implements :func:`~eventsourcing.projection.Projection.process_event`
-by calling :func:`incr_created_event_count` for each :class:`Aggregate.Created <eventsourcing.domain.Aggregate.Created>` event,
-and by calling :func:`incr_subsequent_event_count` for each subsequent :class:`Aggregate.Event <eventsourcing.domain.Aggregate.Event>`.
+The :class:`~eventsourcing.projection.Projection` class is a generic class because it has one type variable,
+which is expected to be a type of tracking recorder.
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: CountProjection
+The :class:`~eventsourcing.projection.Projection` class has one required constructor argument, :data:`tracking_recorder`,
+which is expected to be a tracking recorder object of the type specified by the type variable. The constructor argument
+is used to initialise an object attribute called :data:`tracking_recorder`.
 
-.. _Projection runner:
+The :class:`~eventsourcing.projection.Projection` class is an abstract class because it defines an abstract method,
+:func:`~eventsourcing.projection.Projection.process_event`, that must be implemented by subclasses.
 
-Projection Runner
-=================
+The intention of this class is that it will be subclassed, and that domain events of an application will be processed by
+calling an implementation of the :func:`~eventsourcing.projection.Projection.process_event`, which will call command
+methods on a tracking recorder object given as the constructor argument when the subclass is constructed.
+
+.. code-block:: python
+
+    from abc import ABC, abstractmethod
+    from eventsourcing.domain import DomainEventProtocol
+    from eventsourcing.dispatch import singledispatchmethod
+    from eventsourcing.persistence import Tracking, TrackingRecorder
+    from eventsourcing.popo import POPOTrackingRecorder
+    from eventsourcing.postgres import PostgresTrackingRecorder
+    from eventsourcing.projection import Projection
+    from eventsourcing.sqlite import SQLiteTrackingRecorder
 
-The library's :class:`~eventsourcing.projection.ProjectionRunner` class is provided for the purpose
-or running projections.
+    class MyTrackingRecorderInterface(TrackingRecorder, ABC):
+        @abstractmethod
+        def my_command(self, tracking: Tracking):
+            """Updates materialised view"""
 
-A projection runner object can be constructed with an application class, a projection class, a tracking
-recorder class, and an environment that specifies the persistence modules to be used by the application
-and the tracking recorder.
+    class MyPOPOTrackingRecorder(MyTrackingRecorderInterface, POPOTrackingRecorder):
+        def my_command(self, tracking: Tracking):
+            with self._datastore:
+                # Insert tracking record...
+                self._insert_tracking(tracking)
+                # ...and then update materialised view.
 
-The projection runner will construct an instance of the given application class, and an instance of
-the given projection class, and an instance of the given tracking recorder class. It will
-:ref:`subscribe to its application <Subscriptions>`, from the position indicated by its tracking recorder's
-:func:`~eventsourcing.persistence.TrackingRecorder.max_tracking_id` method. And then it will call
-the :func:`~eventsourcing.projection.Projection.process_event` method of the projection for each event
-in the application sequence.
+    class MySQLiteTrackingRecorder(MyTrackingRecorderInterface, SQLiteTrackingRecorder):
+        def my_command(self, tracking: Tracking):
+            ...
 
-Because it starts a :ref:`subscription <Subscriptions>` to the application, it will first catch up by
-processing already recorded events that have not yet been processed. And then it will continue indefinitely
-to process events that are recorded after the runner has been started.
+    class MyPostgresTrackingRecorder(MyTrackingRecorderInterface, PostgresTrackingRecorder):
+        def my_command(self, tracking: Tracking):
+            ...
 
-The :class:`~eventsourcing.projection.ProjectionRunner` class has a :func:`~eventsourcing.projection.ProjectionRunner.run_forever`
-method, which blocks indefinitely, or until an optional timeout, or until an exception is raised by the projection or
-by the subscription. This allows an event processing component to be started and run independently as a
-separate operating system process, and then to terminate when there is an error. Operators of the system can
-examine the error and resume processing by reconstructing the runner. Some errors may be transient operational
-issues, such as database connectivity, in which case the processing could be resumed automatically. Some errors
-may be programming errors, and will require manual intervention before the event processing can continue.
+    class MyProjection(Projection[MyTrackingRecorderInterface]):
+        @singledispatchmethod
+        def process_event(self, domain_event: DomainEventProtocol, tracking: Tracking) -> None:
+            pass
 
-The :class:`TestCountProjection` class shown below constructs a :class:`~eventsourcing.projection.ProjectionRunner`
-with the library's :class:`~eventsourcing.application.Application` class, the :class:`CountProjection` class,
-and the :class:`POPOCountRecorder`.
+        @process_event.register
+        def _(self, domain_event: Aggregate.Event, tracking: Tracking) -> None:
+            self.tracking_recorder.my_command(tracking)
 
-Two aggregates are saved in the "write model". They have two subsequent events each.
-The total counts for the application events are obtained from the "read model".
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: TestCountProjection
+.. _Projection runner:
+
+Projection runner
+=================
+
+This module provides a :class:`~eventsourcing.projection.ProjectionRunner` class, which can be used to run projections.
 
-If the application "write model" and the tracking recorder "read model" use a durable database, such as
-PostgreSQL, any instance of the application can be used to write events, and any instance of the tracking
-recorder can be used to query the materialised view. However, in this case, using the :ref:`POPO module <popo-module>`
-means that we need to use the same instance of the application and of the recorder.
+Projection runner objects can be constructed by calling the class with an application class, a projection class,
+and a tracking recorder class, and an optional environment. An application object will be constructed using the
+application class and the environment. An infrastructure factory will be constructed for the tracking recorder,
+also using the environment. A projection will also be constructed using the tracking recorder.
 
+The projection runner will then start a subscription to the application, from the position indicated by the tracking
+recorder's :func:`~eventsourcing.persistence.TrackingRecorder.max_tracking_id` method.
 
-With PostgreSQL
-===============
+The projection runner will iterate over the application subscription, calling the projection's
+:func:`~eventsourcing.projection.Projection.process_event` method for each domain event
+and tracking object yielded by the application subscription.
 
-We can also implement the tracking recorder to work with PostgreSQL. The :func:`_incr_counter` method
-of :class:`PostgresCountRecorder`, included below, updates the materialised view and records
-a tracking object atomically in the same database transaction.
+The projection runner has a method :func:`~eventsourcing.projection.Projection.run_forever` which will block
+until either the :func:`~eventsourcing.projection.Projection.process_event` method raises an error, or until
+the application subscription raises an error, or until the optional timeout is reached.
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: PostgresCountRecorder
+Projection runner objects can be used as context managers.
 
-Because this example uses a durable database, separate instances of the application and the recorder
-can be used as interfaces to the "write model" and the "read model".
+.. code-block:: python
 
-The application and the projection could use separate databases, but in this example they simply
-use different tables in the same database.
+    from eventsourcing.projection import ProjectionRunner
 
+    with ProjectionRunner(
+        application_class=Application,
+        projection_class=MyProjection,
+        tracking_recorder_class=MyPOPOTrackingRecorder,
+        env={},
+    ) as runner:
+        runner.run_forever(timeout=1)
 
-.. literalinclude:: ../../tests/projection_tests/test_projection.py
-    :pyobject: TestCountProjectionWithPostgres
 
-See example :doc:`/topics/examples/fts-projection` for a more substantial example.
+See :doc:`Tutorial - Part 4 </topics/tutorial/part4>` for more guidance on using this module.
 
 Code reference
 ==============

docs/topics/tutorial.rst

@@ -17,3 +17,4 @@ documentation.
    tutorial/part2
    tutorial/part3
    tutorial/part4
+   tutorial/part5

docs/topics/tutorial/part1.rst

@@ -2,7 +2,7 @@
 Tutorial - Part 1 - Getting started
 ===================================
 
-Part 1 of this :doc:`tutorial </topics/tutorial>` introduces the library's
+This part the :doc:`tutorial </topics/tutorial>` introduces the library's
 :class:`~eventsourcing.domain.Aggregate` and :class:`~eventsourcing.application.Application`
 classes, showing and explaining how they can be used together to
 write an event-sourced application in Python.

docs/topics/tutorial/part2.rst

@@ -2,6 +2,8 @@
 Tutorial - Part 2 - Aggregates
 ==============================
 
+This part of the tutorial shows in more detail how to define event-sourced aggregates.
+
 In :doc:`Part 1 </topics/tutorial/part1>` we learned
 how to write event-sourced aggregates and applications
 in Python.

docs/topics/tutorial/part3.rst

@@ -2,6 +2,7 @@
 Tutorial - Part 3 - Applications
 ================================
 
+This part of the tutorial shows in more detail how to define event-sourced applications.
 
 As we saw in :doc:`Part 1 </topics/tutorial/part1>`, we can
 use the library's :class:`~eventsourcing.application.Application` class to define event-sourced