Design Document

ZMF Design Document

This document gives an overview about the basic architecture of the Zero Module Framework (ZMF) and its components. For a detailed documentation see Doxygen documentation of ZMF or Javadoc documentation of JMF.

ZMF is a framework to build systems of distributed modules connected by a ZMQ message bus. Any module runs on a seperate ZMF instance. Modules can communicate with other modules or access states of other modules using the ZMF framework.

ZMF Architecture

The ZMF framework consists of four major parts: ZmfCore, PeerDiscoveryService, MessagingService and EventDispatcher.

ZmfCore connects all internal components and offers a public interface. It controls the module lifecycle and controls and performs the framework startup/shutdown and module enable/disable. A module only enables or disables when enable/disable is initiated by the ZmfCore. It also manages the modules dependencies. A module can only enable if all dependencies are satisfied and it will be disabled when not.

A ZMF module is a class implementing AbstractModule and thereby at least the methods enable and disable. A ZMF instance is an application based on ZMF which is running a module. At startup of a ZMF instance, the module instance is passed as parameter to the ZmfCore. Then the ZmfCore takes control over the module and initiates the startup.

All public access goes through the ZmfCore. Therefore it offers two interfaces: IZmfInstanceAccess and IZmfInstanceController.

IZmfInstanceAccess allows modules to access ZMF functionalities: communication with other modules (publish/subscribe, request/reply) or the PeerRegistry (other modules and their states).

    virtual const std::shared_ptr<zmf::config::IConfigurationProvider>& getConfigurationProvider()
    virtual const std::shared_ptr<zmf::discovery::PeerRegistry>& getPeerRegistry()
    virtual zmf::data::ZmfInReply sendRequest(const zmf::data::ModuleUniqueId& target, const zmf::data::ZmfMessage& msg)
    virtual IZmfInstanceAccess::SubscriptionHandle subscribe(const zmf::data::MessageType& topic, std::function<void(const zmf::data::ZmfMessage& msg, const zmf::data::ModuleUniqueId& sender)> handler)
    virtual void publish(const zmf::data::ZmfMessage& msg)
    virtual void onModuleAdditionalStateChanged(std::vector<uint8_t> additionalState)

IZmfInstanceController allows to initiate module enabling or disabling and shutdown of the whole ZMF instance or other remote instances.

    virtual void requestEnableModule()
    virtual void requestDisableModule()
    virtual void requestStopInstance()
    virtual void requestEnableRemoteInstance()
    virtual void requestDisableRemoteInstance()
    virtual void requestStopRemoteInstance()
    virtual void stopInstance()

PeerDiscoveryService collects states of other modules in the network and registers them in the PeerRegistry. The PeerRegistry can be accessed by the ZmfCore and the module. MessagingService manages the communication via ZMQ. EventDispatcher dispatches all external events: PeerStatechanges and incoming messages (requests and events). PeerDiscoveryService, MessagingService and EventDispatcher are described in detail below.

Configuration System

ZMF has a built in configuration system. At startup the path of a configuration file can be specified. If not default values will be used. A configuration file is a text file in Poco/Java properties format: Line wise key-value pairs (e.g. “ZMF_INMSG_BUFFER_SIZE = 100000”).

At runtime, configuration values can be accessed by the IConfigurationProvider interface. It offers getters for String, Boolean, Integer and Double values.

Users can define their own configuration values in the config file and access them at runtime.

In addition there are ZMF configuration parameters:

Name	Type	Unit/Format	Default	Description
ZMF
ZMF_INMSG_BUFFER_SIZE	Int32	Messages	100,000	Maximum size of the inbound message buffer of ZMF, 0 for infinite
logging.loggers.root.level	String			Poco loglevel (eg. "INFORMATION")
ZMQ
ZMF_NETWORK_INTERFACE_NAME	String	IP?	""	Network interface to be used for ZMQ. Automatically chooses Interface if ""
ZMF_ZMQ_ZMQ_RCVBUF	Int32	Bytes	0	Set the ZMQ underlying kernel receive buffe, 0 for system default
ZMF_ZMQ_ZMQ_RCVHWM	Int32	Messages	100,000	ZMQ high water mark for inbound messages, 0 for infinite
ZMF_ZMQ_ZMQ_SNDBUF	Int32	Bytes	0	Set the ZMQ underlying kernel transmit buffer, 0 for system default
ZMF_ZMQ_ZMQ_SNDHWM	Int32	Messages	100,000	ZMQ high water mark for outbound messages, 0 for infinite

Module Execution

A module's execution is controlled by the ZmfCore. A module can be active or inactive. When a module is in the state active it can interact with other modules and all ZMF features are active. When the module is in the inactive state it is in a standby mode. All module threads must be suspended and the ZMF is in a standby mode. The module can then be activated by requesting enable.

After startup there are three available actions: requestEnable, requestDisable and requestStop. All three can be performed locally by IZmfInstanceController or remote by an other module.

The following diagram shows an example of a ZMF execution:

Startup

When running the ZMF run method, first other basic functionalities (such as ZMQ connections) are initialized. After finishing this, the instance is in the state "inactive". The module will stay inactive as long as not all dependencies are satisfied and no autoenable or enable request active. When the module was started with autoEnable it will start enabling immediately if possible.

In Short:

1. Starting method (eg. main method) creates module class
2. Starting method starts ZMF run method
3. Basic initalization (ZMQ etc.)
4. 1. Termination if inizialization fails
5. Inactive state - waiting for all dependencies resolved and enable requests
6. Enabling
7. 1. Terminate if Enable fails and exitOnEnableFail flag set
8. 2. Stay inactive when Enable fails and enable requested
9. Enabled state when Enable was successful

Enable

ZMF tries to enable a module if enable is requested, the module is inactive and all dependencies are satisfied. Enable (if no auto enable) can be initiated by calling requestEnableModule. This can be done locally using the IZmfInstanceController interface or remote by an other module. After disabling a module will be in the state active.

Disable

A module will be disabled by the ZmfCore when requested - locally by IZmfInstanceController or remote by an other module. After disabling a module will be in the state inactive.

Stop

The stop commands shuts down the whole ZMF instance, the application will be terminated. If the module is enabled it will be disabled first. After disabling all ZMF services will be shut down. Stop is irreversible, after stopping the application will be terminated.

PeerDiscovery-Service

The PeerDiscoveryService is an essential part of the ZMF which helps to recognize other ZSDN-modules in the network. It is started and stopped from the ZmfCore at the same time the ZmfInstance is started or stopped. The class itself initializes a thread which continously sends out so called heart-beats to the network and also receives those of other modules at the same time. These heartbeats contain the actual state of the own module. The messages are received by the PeerDiscoveryServices of other modules in the network and tell them about this module. The communication is handled via google protobuffer messages that contain all necessary state information. All messages contain the following information:

message-format:

Field Name	Description	Required	Type (protobuf)
zmq-pub-port	the portnumber used for zmq message publishings	required	uint32
zmq-rep-port	the portnumber used for zmq message replies	required	uint32
/2.sender-unique-ID	16bit module-type-ID	required	uint32
.64bit module-instance-ID	required	uint64
sender-name	the name of the source from where the message was sent	required	string
module-version	the version that the module has	required	uint32
lifecycle-state	8bit lifecycle state (active,inactive,dead)	required	uint32
lifecycle-additional-info	8bit lifecycle additional information, eg. crash (not in use yet)	optional	uint32
additional-state-info	8bit array additional state information for individual needs	optional	bytes
multicast-identifier	large random number, used to recognize messages sent from the own module	required	unit32

(listed types are those of google protobuffers, they are converted to and from their equivalents in C++ and Java)

They are sent and received as multicasts over an udp socket since every PeerDiscoveryService subscribes to the same multicast group. Every time the multicastThread receives a multicast from the network, the PeerDiscovery is updated with the received information. The PeerDiscovery itself is a data class where all currently known modules are listed with their states. It can be accessed by the own module, for example if it has certain dependencies on other modules which need to be fulfilled before the module itself can be started. If a multicast from an unknown module is received, it is added to the PeerDiscovery. In the same way a module is removed again when either a message is received that tells the module is dead now or no heartbeats were received for a longer time which means that the module timed out and is potentially dead.

Messaging-Service

IZmfMessagingCoreInterface:

    virtual void onSubMsgReceived(const zmf::data::ZmfMessage& message, const zmf::data::ModuleUniqueId& sender) = 0;
    virtual void onRequestMsgReceived(const zmf::messaging::ExternalRequestIdentity id, const zmf::data::ZmfMessage& message, const zmf::data::ModuleUniqueId& sender) = 0;

IZmfMessagingService:

    virtual bool start(IZmfMessagingCoreInterface* const corePtr, std::shared_ptr<zmf::data::ModuleHandle> selfHandle, std::shared_ptr<zmf::config::IConfigurationProvider> config) = 0;
    virtual void stop() = 0;
    virtual void peerJoin(std::shared_ptr<zmf::data::ModuleHandle> module) = 0;
    virtual void peerLeave(std::shared_ptr<zmf::data::ModuleHandle> module) = 0;
    virtual void subscribe(const zmf::data::MessageType& topic) = 0;
    virtual void unsubscribe(const zmf::data::MessageType& topic) = 0;
    virtual void publish(const zmf::data::ZmfMessage& msg) = 0;
    virtual zmf::data::ZmfInReply sendRequest(const zmf::data::ModuleUniqueId& target, const zmf::data::ZmfMessage& msg) = 0;
    virtual void cancelRequest(uint64_t requestID, bool manual) = 0;
    virtual void sendReply(ExternalRequestIdentity requestID, const zmf::data::ZmfMessage& reply) = 0;
    virtual void onDisable() = 0;

The Messaging Service implements the IZmfMessagingService interface using ZMQ-Sockets as underlying communication channel. It is save to call the implementation with multiple threads, any non thread save elements are protected by locks. On the other hand the implementation will only call the IZmfMessagingCoreInterface with the POLLER thread. The Publish/Subscribe functionality is mapped directly to the ZMQ Pub/Sub sockets. The Request/Reply communication is implemented using the Harmony-Pattern described in the ZMQ Guide http://zguide.zeromq.org/php:chapter8#True-Peer-Connectivity-Harmony-Pattern. A ROUTER socket is used as endpoint for incoming connections from DEALER sockets. Additionally A PUSH and a PULL socket pair is used for internal signaling. The port numbers of the PUB and ROUTER Sockets are published the peer registry so peers can connect. As soon as a new peer is announced to the implementation from the peer discovery, it will connect its sub socket to the peers pub socket and will create a new DEALER socket and will connect it to the peers ROUTER socket. Additionally a HELLO message is send containing the following information:

• com type: HELLO
• sender identity: ModuleUniqueId (serialized as zmf::proto::SenderId using Protobuf)
• sender rep addr: string
• sender pub addr: string

When such a HELLO message is received and the sending peer is not yet known, connections as described above are established. This is done to ensure there are no asynchronous connections as those might lead to message loss.

Publish/Subscribe message format:

• Match/Topic: 128bit Message Identifier, dynamic length
• sender identity: ModuleUniqueId (serialized as zmf::proto::SenderId using Protobuf)
• Payload

Request/Reply message format:

• com type: req/rep
• message id: uint64_t unique per sender
• : ModuleUniqueId (serialized as zmf::proto::SenderId using Protobuf)
• message type: MessageType
• payload: ...

Req/Rep handing

In contrast to the pub/sub handling where the implementation simply passes messages through, req/rep messages require more logic. When sendRequest(target, message) is invoked, a new entry in a outstandingReplies-Map is added containing a unique identifier for the message as key and the response object returned to the caller as value. After the map is filled, the DEALER socket for the given target will send the message to the target. If a reply is received, the outstandingReplies-Map is checked if for the message id an entry exists. If not, the reply is discarded, If present, the entry is deleted from the map and the message set on the future in the response object. If a Request message is received, an entry in a outstandingRequests-Map is created consisting of (requestID) -> (senderID). Now the request is passed to IZmfMessagingCoreInterface. If sendReply(requestID, message) is called, the map is checked and if a entry exists, a reply is send back.

Event Dispatcher

The Event Dispatcher implements the IZmfMessagingCoreInterface and wraps the MessagingService from the ZMF-Core. It is also responsible to deliver Peer-State-Changes, Incoming Requests and Publish-Events to the Module. The implementation assures that the module code is only accessed by a single thread at any time. The Event Dispatcher also ensures that the module is not flooded with pub/sub events and will drop incoming pub/sub events if the delivery queue is full. Another task of the event dispatcher is to handle System_requests that are used for remote enable/disable/stop.