General Specifications
node-red-biglib is a library module for my big nodes projects. These big nodes are replacement or new nodes for node-red. They all have multiple features sur as
- designed to work with blocks of data that means big volume. This feature implies control messages as a csv parser needs to know where is the first line while receiving multiple blocks
- visually able to give their status as working with a lot of data may takes some time
- always have a second output for control messages. These messages are used to tell other big nodes what's happening but also to give statistics while running such as duration, start time, number of records, actual configuration and so on
- They all are able to work as filters (i.e. they receive data, they transform data and they send data) or as a generator (i.e. they are the starting point of a data feed)
- They may be configured on the fly
node is made to work with messages, payloads. Each message must be an entire feed of data, a file for example. With high volumes, this behaviour sometimes turns into out of memory messages or service hangs.
Working with blocks implies that the node knows when the feed starts and when it ends. The control messages have two purposes, to give statistics and to give other nodes the informations about start and stop.
Example of bindings for a big file reader and a big csv parser
The second connector is bound to the big csv entry so that it knows where is the first line and so the header
If the node receives a message with { "control": { "state": "start" } }, the node turns into block mode. In this mode, each payload even with the control message is a block of data it will parse.
When the node receives a message with { "control": { "state": "end" } }, the node parses the payload if defined and ends the work sending also a control message with an end status.
The node may be used as a classic filter to filter or parse a simple coherent block of data within a single payload message.
If a payload message is sent to the node, it acts as a filter in "standalone" mode as if has received a all in one bundle with a start + data + end flow.
Big nodes may acts as data generators usually from files. If a big csv node is setup with a filename, any incoming payload message is a trigger to start reading the filename. The incoming payload message is so not a data block to parse any more.
If filenames are sent by a watch node for example, the best way to parse them is to use a big file node:
Big nodes may receive config messages in order to change their internal behaviour. For example, a big line node has a keepEmptyLines option available through the setup window. If a node receives a message with { "config": { "keepEmptyLines": true } }, il will reconfigure itself on the fly.
If such a message contains a filename to parse, the big node turns itself into a data generator as described before