Interface data storage in PROGMEM (#71)

Adds a webpack plugin to package interface as PROGMEM into a header file in the framework.
Adds a build flag to optionally enable serving from PROGMEM or SPIFFS as required
Adds documentation changes to describe changes

Author: rjwats

.gitignore

@@ -1,10 +1,9 @@
 .pio
-.pioenvs
-.piolibdeps
 .clang_complete
 .gcc-flags.json
 *Thumbs.db
 /data/www
+/lib/framework/WWWData.h
 /interface/build
 /interface/node_modules
 .vscode

.travis.yml

@@ -1,6 +1,10 @@
 language: python
 python:
-    - "2.7"
+    - "3.8"
+
+before_install:
+  - nvm install 10.15.3
+  - nvm use 10.15.3
 
 sudo: false
 cache:

README.md

@@ -30,7 +30,6 @@ You will need the following before you can get started.
 
 * [PlatformIO](https://platformio.org/) - IDE for development
 * [Node.js](https://nodejs.org) - For building the interface with npm
-* Bash shell, or [Git Bash](https://gitforwindows.org/) if you are under windows
 
 ### Building and uploading the firmware
 
@@ -74,53 +73,61 @@ Alternatively run the 'upload' target:
 platformio run -t upload
 ```
 
-### Building the interface
+### Building & uploading the interface
 
 The interface has been configured with create-react-app and react-app-rewired so the build can customized for the target device. The large artefacts are gzipped and source maps and service worker are excluded from the production build. This reduces the production build to around ~200k, which easily fits on the device.
 
-Change to the ['interface'](interface) directory with your bash shell (or Git Bash) and use the standard commands you would with any react app built with create-react-app:
-
-#### Change to interface directory
+The interface will be automatically built by PlatformIO before it builds the firmware. The project can be configured to serve the interface from either SPIFFS or PROGMEM as your project requires. The default configuration is to serve the content from SPIFFS which requires an additional upload step which is documented below.
 
-```bash
-cd interface
-```
+#### Uploading the file system image
 
-#### Download and install the node modules
+If service content from SPIFFS (default), build the project first. Then the compiled interface may be uploaded to the device by pressing the "Upload File System image" button:
 
-```bash
-npm install
-```
+![uploadfs](/media/uploadfs.png?raw=true "uploadfs")
 
-#### Build the interface
+Alternatively run the 'uploadfs' target:
 
 ```bash
-npm run build
+platformio run -t uploadfs
 ```
 
-> **Note**: The build command will also delete the previously built interface, in the ['data/www'](data/www) directory, replacing it with the freshly built one ready to upload to the device.
+#### Serving the interface from PROGMEM
 
-#### Uploading the file system image
+You can configure the project to serve the interface from PROGMEM by uncommenting the -D PROGMEM_WWW build flag in ['platformio.ini'](platformio.ini) then re-building and uploading the firmware to the device. 
 
-The compiled user interface may be uploaded to the device by pressing the "Upload File System image" button:
+Be aware that this will consume ~150k of program space which can be especially problematic if you already have a large build artefact or if you have added large javascript dependencies to the interface. The ESP32 binaries are large already, so this will be a problem if you are using one of these devices and require this type of setup.
 
-![uploadfs](/media/uploadfs.png?raw=true "uploadfs")
+A method for working around this issue can be to reduce the amount of space allocated to SPIFFS by configuring the device to use a differnt strategy partitioning. If you don't require SPIFFS other than for storing config one approach might be to configure a minimal SPIFFS partition.
 
-Alternatively run the 'uploadfs' target:
+For a ESP32 (4mb variant) there is a handy "min_spiffs.csv" partition table which can be enabled easily:
 
-```bash
-platformio run -t uploadfs
+```yaml
+[env:node32s]
+board_build.partitions = min_spiffs.csv
+platform = espressif32
+board = node32s
 ```
 
+This is largley left as an exersise for the reader as everyone's requirements will vary.
+
 ### Running the interface locally
 
-You can run a local development server to allow you preview changes to the front end without the need to upload a file system image to the device after each change. Change to the interface directory and run the following command:
+You can run a local development server to allow you preview changes to the front end without the need to upload a file system image to the device after each change. 
+
+Change to the ['interface'](interface) directory with your bash shell (or Git Bash) and use the standard commands you would with any react app built with create-react-app:
+
+```bash
+cd interface
+```
+
+Install the npm dependencies, if required and start the development server:
 
 ```bash
+npm install
 npm start
 ```
 
-> **Note**: To run the interface locally you will need to modify the endpoint root path and enable CORS.
+> **Note**: To run the interface locally you may need to modify the endpoint root path and enable CORS.
 
 #### Changing the endpoint root
 
@@ -141,7 +148,9 @@ You can enable CORS on the back end by uncommenting the -D ENABLE_CORS build fla
 
 ## Device Configuration
 
-As well as containing the interface, the SPIFFS image (in the ['data'](data) folder) contains a JSON settings file for each of the configurable features. The config files can be found in the ['data/config'](data/config) directory:
+The SPIFFS image (in the ['data'](data) folder) contains a JSON settings file for each of the configurable features. 
+
+The config files can be found in the ['data/config'](data/config) directory:
 
 File | Description
 ---- | -----------
@@ -173,30 +182,31 @@ It is recommended that you change the JWT secret and user credentials from their
 
 This project supports ESP8266 and ESP32 platforms. To support OTA programming, enough free space to upload the new sketch and file system image will be required. It is recommended that a board with at least 2mb of flash is used.
 
-By default, the target device is "esp12e". This is a common ESP8266 variant with 4mb of flash:
+The pre-configured environments are "esp12e" and "node32s". These are common ESP8266/ESP32 variants with 4mb of flash:
 
-![ESP12E](/media/esp12e.jpg?raw=true "ESP12E")
+![ESP12E](/media/esp12e.jpg?raw=true "ESP12E") ![ESP32](/media/esp32.jpg?raw=true "ESP32")
 
-The settings file ['platformio.ini'](platformio.ini) configures the platform and board:
+The settings file ['platformio.ini'](platformio.ini) configures the supported environments. Modify these, or add new environments for the devides you need to support. The default environments are as follows:
 
-```
+```yaml
 [env:esp12e]
 platform = espressif8266
 board = esp12e
-```
-
-If you want to build for an ESP32 device, all you need to do is re-configure ['platformio.ini'](platformio.ini) with your devices settings. 
-
-![ESP32](/media/esp32.jpg?raw=true "ESP32")
-
-Building for the common esp32 "node32s" board for example requires the following configuration:
+board_build.f_cpu = 160000000L
 
-```
 [env:node32s]
 platform = espressif32
 board = node32s
 ```
 
+If you want to build for a different device, all you need to do is re-configure ['platformio.ini'](platformio.ini) and select an alternative environment by modifying the default_envs variable. Building for the common esp32 "node32s" board for example:
+
+```yaml
+[platformio]
+;default_envs = esp12e
+default_envs = node32s
+```
+
 ## Customizing and theming
 
 The framework, and MaterialUI allows for a reasonable degree of customization with little effort.
@@ -274,7 +284,11 @@ void setup() {
   Serial.begin(SERIAL_BAUD_RATE);
 
   // start the file system (must be done before starting the framework)
+#ifdef ESP32
+  SPIFFS.begin(true);
+#elif defined(ESP8266)
   SPIFFS.begin();
+#endif
 
   // start the framework and demo project
   esp8266React.begin();

interface/config-overrides.js

@@ -1,7 +1,8 @@
 const ManifestPlugin = require('webpack-manifest-plugin');
 const WorkboxWebpackPlugin = require('workbox-webpack-plugin');
 const MiniCssExtractPlugin = require('mini-css-extract-plugin');
-const CompressionPlugin = require("compression-webpack-plugin");
+const CompressionPlugin = require('compression-webpack-plugin');
+const ProgmemGenerator = require('./progmem-generator.js');
 
 const path = require('path');
 const fs = require('fs');
@@ -21,6 +22,9 @@ module.exports = function override(config, env) {
     miniCssExtractPlugin.options.filename = "css/[id].[contenthash:4].css";
     miniCssExtractPlugin.options.chunkFilename = "css/[id].[contenthash:4].c.css";
 
+    // build progmem data files
+    config.plugins.push(new ProgmemGenerator({ outputPath: "../lib/framework/WWWData.h", bytesPerLine: 20 }));
+
     // add compression plugin, compress javascript
     config.plugins.push(new CompressionPlugin({
       filename: "[path].gz[query]",

interface/package-lock.json

@@ -7,6 +7,7 @@
     "@material-ui/icons": "^4.5.1",
     "compression-webpack-plugin": "^2.0.0",
     "jwt-decode": "^2.2.0",
+    "mime-types": "^2.1.25",
     "moment": "^2.24.0",
     "notistack": "^0.9.6",
     "prop-types": "^15.7.2",
@@ -17,11 +18,12 @@
     "react-material-ui-form-validator": "^2.0.9",
     "react-router": "^5.1.1",
     "react-router-dom": "^5.1.1",
-    "react-scripts": "3.0.1"
+    "react-scripts": "3.0.1",
+    "zlib": "^1.0.5"
   },
   "scripts": {
     "start": "react-app-rewired start",
-    "build": "react-app-rewired build && rm -rf ../data/www && cp -r build ../data/www",
+    "build": "react-app-rewired build",
     "test": "react-app-rewired test --env=jsdom",
     "eject": "react-scripts eject"
   },

interface/package.json

@@ -0,0 +1,122 @@
+const { resolve, relative, sep } = require('path');
+const { readdirSync, existsSync, unlinkSync, readFileSync, createWriteStream } = require('fs');
+var zlib = require('zlib');
+var mime = require('mime-types');
+
+const ARDUINO_INCLUDES = "#include <Arduino.h>\n\n";
+
+function getFilesSync(dir, files = []) {
+  readdirSync(dir, { withFileTypes: true }).forEach(entry => {
+    const entryPath = resolve(dir, entry.name);
+    if (entry.isDirectory()) {
+      getFilesSync(entryPath, files);
+    } else {
+      files.push(entryPath);
+    }
+  })
+  return files;
+}
+
+function coherseToBuffer(input) {
+  return Buffer.isBuffer(input) ? input : Buffer.from(input);
+}
+
+function cleanAndOpen(path) {
+  if (existsSync(path)) {
+    unlinkSync(path);
+  }
+  return createWriteStream(path, { flags: "w+" });
+}
+
+class ProgmemGenerator {
+
+  constructor(options = {}) {
+    const { outputPath, bytesPerLine = 20, indent = "  ", includes = ARDUINO_INCLUDES } = options;
+    this.options = { outputPath, bytesPerLine, indent, includes };
+  }
+
+  apply(compiler) {
+    compiler.hooks.emit.tapAsync(
+      { name: 'ProgmemGenerator' },
+      (compilation, callback) => {
+        const { outputPath, bytesPerLine, indent, includes } = this.options;
+        const fileInfo = [];
+        const writeStream = cleanAndOpen(resolve(compilation.options.context, outputPath));
+        try {
+          const writeIncludes = () => {
+            writeStream.write(includes);
+          }
+
+          const writeFile = (relativeFilePath, buffer) => {
+            const variable = "ESP_REACT_DATA_" + fileInfo.length;
+            const mimeType = mime.lookup(relativeFilePath);
+            var size = 0;
+            writeStream.write("const uint8_t " + variable + "[] PROGMEM = {");
+            const zipBuffer = zlib.gzipSync(buffer);
+            zipBuffer.forEach((b) => {
+              if (!(size % bytesPerLine)) {
+                writeStream.write("\n");
+                writeStream.write(indent);
+              }
+              writeStream.write("0x" + ("00" + b.toString(16).toUpperCase()).substr(-2) + ",");
+              size++;
+            });
+            if (size % bytesPerLine) {
+              writeStream.write("\n");
+            }
+            writeStream.write("};\n\n");
+            fileInfo.push({
+              uri: '/' + relativeFilePath.replace(sep, '/'),
+              mimeType,
+              variable,
+              size
+            });
+          };
+
+          const writeFiles = () => {
+            // process static files
+            const buildPath = compilation.options.output.path;
+            for (const filePath of getFilesSync(buildPath)) {
+              const readStream = readFileSync(filePath);
+              const relativeFilePath = relative(buildPath, filePath);
+              writeFile(relativeFilePath, readStream);
+            }
+            // process assets
+            const { assets } = compilation;
+            Object.keys(assets).forEach((relativeFilePath) => {
+              writeFile(relativeFilePath, coherseToBuffer(assets[relativeFilePath].source()));
+            });
+          }
+
+          const generateWWWClass = () => {
+            return `typedef std::function<void(const String& uri, const String& contentType, const uint8_t * content, size_t len)> RouteRegistrationHandler;          
+
+class WWWData {
+${indent}public:
+${indent.repeat(2)}static void registerRoutes(RouteRegistrationHandler handler) {
+${fileInfo.map(file => `${indent.repeat(3)}handler("${file.uri}", "${file.mimeType}", ${file.variable}, ${file.size});`).join('\n')}
+${indent.repeat(2)}}
+};
+`;
+          }
+
+          const writeWWWClass = () => {
+            writeStream.write(generateWWWClass());
+          }
+
+          writeIncludes();
+          writeFiles();
+          writeWWWClass();
+
+          writeStream.on('finish', () => {
+            callback();
+          });
+        } finally {
+          writeStream.end();
+        }
+      }
+    );
+  }
+}
+
+module.exports = ProgmemGenerator;