Merge pull request #312 from garlick/control_messages

rename keepalives to control messages

Author: mergify[bot]

spec_3.rst

@@ -30,7 +30,7 @@ Goals
 -----
 
 The Flux message protocol v1 provides a way for Flux utilities and services to
-communicate with one another within the context of a job. It has
+communicate with one another within the context of a flux instance. It has
 the following specific goals:
 
 -  Endpoint-count scalability (e.g. to 100K nodes) through multi-hop
@@ -58,10 +58,10 @@ Background
 ``flux-broker`` is a message broker daemon for the Flux resource manager
 framework. A Flux *instance* is a set of interconnected ``flux-broker`` tasks
 that together provide a shared communications substrate for distributed
-resource manager services within a job. Services and utilities communicate
-by passing messages through the session brokers. There are four
-types of messages: events, requests, responses, and keepalives, which
-share a common structure described herein.
+resource manager services. Services and utilities communicate by passing
+messages through the session brokers. There are four types of messages:
+events, requests, responses, and control messages, which share a common
+structure described herein.
 
 Event messages are published such that they are available to subscribers
 throughout the instance. Events are published with a *topic string*
@@ -77,8 +77,9 @@ Responses are optional replies to requests. They follow the ZeroMQ
 ROUTER-DEALER message flow, which unwinds the source address route
 accumulated by the request, and uses them to select among peers at each hop.
 
-Keepalives are control messages used by one peer to indicate to another
-peer that it is still alive when it is not otherwise communicating.
+Control messages are used for connection management and status communication
+between brokers.  Unlike the other message types, they are only used between
+directly connected peers, never routed.
 
 
 Implementation
@@ -242,13 +243,13 @@ ABNF grammar [#f2]_
 
    message       = C:request *S:response
                    / S:event
-                   / C:keepalive
+                   / C:control
 
    ; Multi-part ZeroMQ messages
    C:request       = [routing] topic [payload] PROTO
    S:response      = [routing] topic [payload] PROTO
    S:event         = [routing] topic [payload] PROTO
-   C:keepalive     = PROTO
+   C:control       = PROTO
 
    ; Route frame stack, ZeroMQ DEALER-ROUTER format
    routing         = *identity delimiter
@@ -262,12 +263,12 @@ ABNF grammar [#f2]_
    payload         = *OCTET        ; payload ZeroMQ frame
 
    ; Protocol frame
-   PROTO           = request / response / event / keepalive
+   PROTO           = request / response / event / control
 
    request         = magic version %x01 flags userid rolemask nodeid   matchtag
    response        = magic version %x02 flags userid rolemask errnum   matchtag
    event           = magic version %x04 flags userid rolemask sequence unused
-   keepalive       = magic version %x08 flags userid rolemask errnum   status
+   control         = magic version %x08 flags userid rolemask type     status
 
    ; Constants
    magic           = %x8E          ; magic cookie
@@ -306,9 +307,15 @@ ABNF grammar [#f2]_
    ; Monotonic sequence number in network byte order
    sequence        = 4OCTET
 
+   ; Control message type
+   type            = 4OCTET
+
+   ; Control message status
+   status          = 4OCTET
+
    ; unused 4-byte field
    unused          = %x00.00.00.00
 
 .. [#f1] `RFC 7159: The JavaScript Object Notation (JSON) Data Interchange Format <https://www.rfc-editor.org/rfc/rfc7159.txt>`__, T. Bray, Google, Inc, March 2014.
 
-.. [#f2] For convenience: the ``C:request``, ``S:response``, ``S:event``, and ``C:keepalive`` ABNF non-terminals refer to ZeroMQ messages, sent by client or server, and built from ordered ZeroMQ message parts (frames). Other non-terminals are built from concatenated ABNF terminals per usual. Thus it is meaningful for ``delimiter``, a message frame, to have zero length, since a zero-length message frame is valid ZMTP.
+.. [#f2] For convenience: the ``C:request``, ``S:response``, ``S:event``, and ``C:control`` ABNF non-terminals refer to ZeroMQ messages, sent by client or server, and built from ordered ZeroMQ message parts (frames). Other non-terminals are built from concatenated ABNF terminals per usual. Thus it is meaningful for ``delimiter``, a message frame, to have zero length, since a zero-length message frame is valid ZMTP.

spec_5.rst

@@ -64,9 +64,9 @@ message handlers for its methods, then run the flux reactor. It should
 use event driven (reactive) programming techniques to remain responsive
 while juggling work from multiple clients.
 
-Keepalive messages are sent to the broker via pre-registered reactor
+Status messages are sent to the broker via pre-registered reactor
 watchers to indicate when the module is initializing, running, finalizing,
-or exited. At initialization, a module MAY also manually send a keepalive
+or exited. At initialization, a module MAY also manually send a status
 message to indicate to the broker when initialization is complete. This
 provides synchronization to the broker module loader as well as useful
 runtime debug information that can be reported by ``flux module list``.
@@ -91,35 +91,59 @@ A broker module SHALL export the following global symbols:
    type of error on failure.
 
 
-Keepalive Values
-~~~~~~~~~~~~~~~~
+Status Messages
+~~~~~~~~~~~~~~~
 
-A broker module SHALL send RFC 3 keepalive messages containing status
-integers to the broker over its broker handle. Status integers are
-enumerated as follows:
+A broker module SHALL be considered to be in one of the following states,
+represented by the integer values shown in parenthesis:
 
 -  FLUX_MODSTATE_INIT (0) - initializing
-
 -  FLUX_MODSTATE_RUNNING (1) - running
-
 -  FLUX_MODSTATE_FINALIZING (2) - finalizing
-
 -  FLUX_MODSTATE_EXITED (3) - ``mod_main()`` exited
 
-Modules SHALL send a keepalive message of ``FLUX_MODSTATE_RUNNING``
-after initialization to notify the broker that the module has started
-successfully. In order to ensure this happens for all modules, A keepalive
-message SHALL be sent via a pre-registered reactor watcher upon a module's
-first entry to the reactor if the module has not otherwise entered the
-RUNNING state. In addition, keepalive messages MAY be sent to the broker
-at regular intervals. The keepalive ``errnum`` field SHALL be zero except
-when ``mod_main()`` returns a value of -1 indicating failure and state
-transitions to FLUX_MODSTATE_EXITED. In this case ``errnum`` SHALL be
-set to the value of POSIX ``errno`` set by ``mod_main()`` before returning.
+Upon loading the module, the broker SHALL initialize the broker state
+to ``FLUX_MODSTATE_INIT``.
+
+After initialization is complete, a module SHALL send an RPC to the
+``broker.module-status`` service with the FLUX_RPC_NORESPONSE flag to
+notify the broker that the module has started successfully.  In order to
+ensure this happens for all modules, the RPC SHALL be sent via a
+pre-registered reactor watcher upon a module's first entry to the reactor
+if the module has not already sent the message.
+
+Example payload:
+
+.. code:: json
+
+   {
+       "status":1
+   }
 
-The broker MAY track the number of session heartbeats since a
-module last sent a message and report this as "idle time"
-for the module.
+After exiting the reactor and before exiting the module thread, the module
+SHALL send an RPC to ``broker.module-status`` indicating that it intends to
+exit.  The module SHALL wait for a response to this message before exiting
+``mod_main()``.
+
+Example payload:
+
+.. code:: json
+
+   {
+       "status":2
+   }
+
+Finally once ``mod_main()`` has exited, the module thread SHALL send an RPC
+to ```broker.module-status`` with the FLUX_RPC_NORESPONSE flag including
+the error status of the module:  zero if ``mod_main()`` exited with a return
+code greater than or equal to zero, otherwise the value of ``errno``.
+
+.. code:: json
+
+   {
+       "status":2,
+       "errnum":0
+   }
 
 
 Load Sequence
@@ -137,10 +161,9 @@ Unload Sequence
 
 The broker module loader SHALL send a ``<service>.shutdown`` request to the
 module when the module loader receives a ``broker.rmmod`` request for the
-module. In response, the broker module SHALL exit ``mod_main()``, send a
-keepalive transition to FLUX_MODSTATE_EXITED state, and exit the
-module’s thread or process. This final state transition indicates to
-the broker that it MAY clean up the module thread.
+module.  In response, the broker module SHALL exit ``mod_main()``, sending
+state transition messages as described above, and exit the module’s thread
+or process. The final state transition indicates to the broker that it MAY clean up the module thread.
 
 
 Built-in Request Handlers

spec_7.rst

@@ -150,7 +150,7 @@ Examples:
        FLUX_MSGTYPE_REQUEST    = 0x01,
        FLUX_MSGTYPE_RESPONSE   = 0x02,
        FLUX_MSGTYPE_EVENT      = 0x04,
-       FLUX_MSGTYPE_KEEPALIVE  = 0x08,
+       FLUX_MSGTYPE_CONTROL    = 0x08,
        FLUX_MSGTYPE_ANY        = 0x0f,
        FLUX_MSGTYPE_MASK       = 0x0f,
    };

spell.en.pws

@@ -79,7 +79,6 @@ dlopen
 dlsym
 insmod
 json
-keepalive
 lsmod
 modstate
 POSIX
@@ -105,7 +104,6 @@ URI
 bitmask
 codec
 crypto
-keepalives
 MSGFLAG
 resize
 scalability