Rework backend add MQTT and WebSocket support

Update back end to add MQTT and WebSocket support
Update demo project to demonstrate MQTT and WebSockets

Author: rjwats

README.md

@@ -273,7 +273,7 @@ There is also a manifest file which contains the app name to use when adding the
 }
 ```
 
-## Back end overview
+## Back end
 
 The back end is a set of REST endpoints hosted by a [ESPAsyncWebServer](https://github.com/me-no-dev/ESPAsyncWebServer) instance. The ['lib/framework'](lib/framework) directory contains the majority of the back end code. The framework contains of a number of useful utility classes which you can use when extending it. The project also comes with a demo project to give you some help getting started. 
 
@@ -328,17 +328,125 @@ void loop() {
 }
 ```
 
-### Adding endpoints
+### Developing with the framework
 
-There are some simple classes that support adding configurable services/features to the device:
+The framework promotes a modular design and exposes features you may re-use to speed up the development of your project. Where possible it is recommended that you use the features the frameworks supplies. These are documented below.
 
-Class | Description
------ | -----------
-[SimpleService.h](lib/framework/SimpleService.h) | Exposes an endpoint to read and write settings as JSON. Extend this class and implement the functions which serialize the settings to/from JSON.
-[SettingsService.h](lib/framework/SettingsService.h) | As above, however this class also handles persisting the settings as JSON to the file system.
-[AdminSettingsService.h](lib/framework/AdminSettingsService.h) | Extends SettingsService to secure the endpoint to administrators only, the authentication predicate can be overridden if required.
+The following diagram visualises how the framework's modular components fit together. They are described in detail below.
 
-The demo project shows how these can be used, explore the framework classes for more examples.
+![framework diagram](/media/framework.png?raw=true "framework diagram")
+
+#### Settings service
+
+The [SettingsService.h](lib/framework/SettingsService.h) class is a responsible for managing settings and interfacing with code which wants to control or respond to changes in those settings. You can define a data class to hold settings then build a SettingsService instance to manage them:
+
+```cpp
+class LightSettings {
+ public:
+  bool on = false;
+  uint8_t brightness = 255;
+};
+
+class LightSettingsService : public SettingsService<LightSettings> {
+};
+```
+
+You may listen for changes to settings by registering an update handler callback. It is possible to remove an update handler later if required. An "origin" pointer is passed to the update handler which may point to the client or object which made the update.
+
+```cpp
+// register an update handler
+update_handler_id_t myUpdateHandler = lightSettingsService.addUpdateHandler(
+  [&](String originId) {
+    Serial.println("The light settings have been updated"); 
+  }
+);
+
+// remove the update handler
+lightSettingsService.removeUpdateHandler(myUpdateHandler);
+```
+
+SettingsService exposes a read function which you may use to safely read the settings. This function takes care of protecting against parallel access to the settings in multi-core enviornments such as the ESP32.
+
+```cpp
+lightSettingsService.read([&](LightSettings& settings) {
+  digitalWrite(LED_PIN, settings.on ? HIGH : LOW)
+});
+```
+
+SettingsService also exposes an update function which allows the caller to update the settings. The update function takes care of calling the registered update handler callbacks once the update is complete.
+
+```cpp
+lightSettingsService.update([&](LightSettings& settings) {
+  settings.on = true;  // turn on the lights!
+});
+```
+
+#### Serialization
+
+When transmitting settings over HTTP, WebSockets or MQTT they must to be marshalled into a serialzable form. The framework uses ArduinoJson for serialization and provides the abstract classes [SettingsSerializer.h](lib/framework/SettingsSerializer.h) and [SettingsDeserializer.h](lib/framework/SettingsDeserializer.h) to facilitate the seriliaztion of settings:
+
+```cpp
+class LightSettingsSerializer : public SettingsSerializer<LightSettings> {
+ public:
+  void serialize(LightSettings& settings, JsonObject root) {
+    root["on"] = settings.on;
+    root["brightness"] = settings.brightness;
+  }
+};
+
+class LightSettingsDeserializer : public SettingsDeserializer<LightSettings> {
+ public:
+  void deserialize(LightSettings& settings, JsonObject root) {
+    settings.on = root["on"] | false;
+    settings.brightness = root["brightness"] | 255;
+  }
+};
+```
+
+It is recommended you make create singletons for your serialzers and that they are stateless:
+
+```cpp
+static LightSettingsSerializer SERIALIZER;
+static LightSettingsDeserializer DESERIALIZER;
+```
+
+#### Endpoints
+
+The framework provides a [SettingsEndpoint.h](lib/framework/SettingsEndpoint.h) class which may be used to register GET and POST handlers to read and update the settings over HTTP. You may construct a SettingsEndpoint as a part of the SettingsService or separately if you prefer. The code below demonstrates how to extend the LightSettingsService class to provide an unsecured endpoint:
+
+```cpp
+class LightSettingsService : public SettingsService<LightSettings> {
+ public:
+  LightSettingsService(AsyncWebServer* server, SecurityManager* securityManager) :
+      _settingsEndpoint(&SERIALIZER, &DESERIALIZER, this, server, "/rest/lightSettings") {
+  }
+
+ private:
+  SettingsEndpoint<LightSettings> _settingsEndpoint;
+};
+```
+
+Endpoint security is provided by authentication predicates which are [documented below](#security-features). A security manager and authentication predicate may be provided if an secure endpoint is required. The demo app shows how endpoints can be secured.
+
+#### Persistence
+
+[SettingsPersistence.h](lib/framework/SettingsPersistence.h) allows you to save settings to the filesystem. SettingsPersistence automatically writes changes to the file system when settings are updated. This feature can be disabled by calling `disableAutomatic()` if manual control of persistence is required.
+
+As with SettingsEndpoint you may elect to construct this as a part of a SettingsService class or separately. The code below demonstrates how to extend the LightSettingsService class to provide persistence:
+
+```cpp
+class LightSettingsService : public SettingsService<LightSettings> {
+ public:
+  LightSettingsService(AsyncWebServer* server, FS* fs, SecurityManager* securityManager) :
+      _settingsEndpoint(&SERIALIZER, &DESERIALIZER, this, server, "/rest/lightSettings"),
+      _settingsPersistence(&SERIALIZER, &DESERIALIZER, this, fs, "/config/lightSettings.json") {
+  }
+
+ private:
+  SettingsEndpoint<LightSettings> _settingsEndpoint;
+  SettingsPersistence<LightSettings> _settingsPersistence;
+};
+```
 
 ### Security features
 

data/config/demoSettings.json

@@ -1,3 +1,3 @@
 {
-  "blink_speed": 100
+  "led_on": true
 }

data/config/mqttSettings.json

@@ -0,0 +1,11 @@
+{
+  "enabled": false,
+  "host": "test.mosquitto.org",
+  "port": 1883,
+  "authenticated": false,
+  "username": "mqttuser",
+  "password": "mqttpassword",
+  "keepAlive": 16,
+  "cleanSession": true,
+  "maxTopicLength": 128
+}

interface/.env.development

@@ -1,3 +1,4 @@
 # Change the IP address to that of your ESP device to enable local development of the UI.
 # Remember to also enable CORS in platformio.ini before uploading the code to the device.
-REACT_APP_ENDPOINT_ROOT=http://192.168.0.21/rest/
+REACT_APP_ENDPOINT_ROOT=http://192.168.0.99/rest/
+REACT_APP_WEB_SOCKET_ROOT=ws://192.168.0.99

interface/package-lock.json

@@ -6,6 +6,7 @@
     "@material-ui/core": "^4.9.8",
     "@material-ui/icons": "^4.9.1",
     "@types/jwt-decode": "^2.2.1",
+    "@types/lodash": "^4.14.149",
     "@types/node": "^12.12.32",
     "@types/react": "^16.9.27",
     "@types/react-dom": "^16.9.5",
@@ -14,6 +15,7 @@
     "@types/react-router-dom": "^5.1.3",
     "compression-webpack-plugin": "^3.0.1",
     "jwt-decode": "^2.2.0",
+    "lodash": "^4.17.15",
     "mime-types": "^2.1.25",
     "moment": "^2.24.0",
     "notistack": "^0.9.7",
@@ -24,6 +26,7 @@
     "react-router": "^5.1.2",
     "react-router-dom": "^5.1.2",
     "react-scripts": "3.4.1",
+    "sockette": "^2.0.6",
     "typescript": "^3.7.5",
     "zlib": "^1.0.5"
   },

interface/package.json

@@ -15,6 +15,7 @@ import Security from './security/Security';
 import System from './system/System';
 
 import { PROJECT_PATH } from './api';
+import MQTT from './mqtt/MQTT';
 
 class AppRouting extends Component {
 
@@ -31,6 +32,7 @@ class AppRouting extends Component {
           <AuthenticatedRoute exact path="/wifi/*" component={WiFiConnection} />         
           <AuthenticatedRoute exact path="/ap/*" component={AccessPoint} />
           <AuthenticatedRoute exact path="/ntp/*" component={NetworkTime} />
+          <AuthenticatedRoute exact path="/mqtt/*" component={MQTT} />
           <AuthenticatedRoute exact path="/security/*" component={Security} /> 
           <AuthenticatedRoute exact path="/system/*" component={System} />          
           <Redirect to="/" />

interface/src/AppRouting.tsx

@@ -9,6 +9,8 @@ export const LIST_NETWORKS_ENDPOINT = ENDPOINT_ROOT + "listNetworks";
 export const WIFI_SETTINGS_ENDPOINT = ENDPOINT_ROOT + "wifiSettings";
 export const WIFI_STATUS_ENDPOINT = ENDPOINT_ROOT + "wifiStatus";
 export const OTA_SETTINGS_ENDPOINT = ENDPOINT_ROOT + "otaSettings";
+export const MQTT_SETTINGS_ENDPOINT = ENDPOINT_ROOT + "mqttSettings";
+export const MQTT_STATUS_ENDPOINT = ENDPOINT_ROOT + "mqttStatus";
 export const SYSTEM_STATUS_ENDPOINT = ENDPOINT_ROOT + "systemStatus";
 export const SIGN_IN_ENDPOINT = ENDPOINT_ROOT + "signIn";
 export const VERIFY_AUTHORIZATION_ENDPOINT = ENDPOINT_ROOT + "verifyAuthorization";

interface/src/api/Endpoints.ts

@@ -1,3 +1,20 @@
 export const PROJECT_NAME = process.env.REACT_APP_PROJECT_NAME!;
 export const PROJECT_PATH = process.env.REACT_APP_PROJECT_PATH!;
 export const ENDPOINT_ROOT = process.env.REACT_APP_ENDPOINT_ROOT!;
+
+// TODO use same approach for rest endpoint?
+export const WEB_SOCKET_ROOT = calculateWebSocketPrefix("/ws");
+
+function calculateWebSocketPrefix(webSocketPath: string) {
+    const webSocketRoot = process.env.REACT_APP_WEB_SOCKET_ROOT;
+    if (!webSocketRoot || webSocketRoot.length === 0) {
+        var loc = window.location, webSocketURI;
+        if (loc.protocol === "https:") {
+            webSocketURI = "wss:";
+        } else {
+            webSocketURI = "ws:";
+        }
+        return webSocketURI + "//" + loc.host + webSocketPath;
+    }
+    return webSocketRoot + webSocketPath;
+}

interface/src/api/Env.ts

@@ -13,6 +13,7 @@ import SettingsIcon from '@material-ui/icons/Settings';
 import AccessTimeIcon from '@material-ui/icons/AccessTime';
 import AccountCircleIcon from '@material-ui/icons/AccountCircle';
 import SettingsInputAntennaIcon from '@material-ui/icons/SettingsInputAntenna';
+import DeviceHubIcon from '@material-ui/icons/DeviceHub';
 import LockIcon from '@material-ui/icons/Lock';
 import MenuIcon from '@material-ui/icons/Menu';
 
@@ -136,6 +137,12 @@ class MenuAppBar extends React.Component<MenuAppBarProps, MenuAppBarState> {
             </ListItemIcon>
             <ListItemText primary="Network Time" />
           </ListItem>
+          <ListItem to='/mqtt/' selected={path.startsWith('/mqtt/')} button component={Link}>
+            <ListItemIcon>
+              <DeviceHubIcon />
+            </ListItemIcon>
+            <ListItemText primary="MQTT" />
+          </ListItem>          
           <ListItem to='/security/' selected={path.startsWith('/security/')} button component={Link} disabled={!authenticatedContext.me.admin}>
             <ListItemIcon>
               <LockIcon />