Skip to content

Commit 526af57

Browse files
tjhowsetjhowse
authored
Add json_key register setting for #9 (#22)
* Reformat readme * Support for json_key register field allowing for multiple values to be published as JSON to a single MQTT topic.
1 parent 051fddf commit 526af57

File tree

5 files changed

+173
-34
lines changed

5 files changed

+173
-34
lines changed

‎README.md‎

Lines changed: 28 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -48,19 +48,16 @@ address_offset: 0
4848
variant: sungrow
4949
scan_batching: 100
5050
```
51-
52-
`ip` (Required) The IP address of the modbus device to be polled. Presently only modbus TCP/IP is supported.
53-
54-
`port` (Optional: default 502) The port on the modbus device to connect to.
55-
56-
`update_rate` (Optional: default 5) The number of seconds between polls of the modbus device.
57-
58-
`address_offset` (Optional: default 0) This offset is applied to every register address to accommodate different Modbus addressing systems. In many Modbus devices the first register is enumerated as 1, other times 0. See section 4.4 of the Modbus spec.
59-
60-
`variant` (Optional) Allows variants of the ModbusTcpClient library to be used. Setting this to 'sungrow' enables support of SungrowModbusTcpClient. This library transparently decrypts the modbus comms with sungrow SH inverters running newer firmware versions.
61-
62-
`scan_batching` (Optional: default 100) Must be between 1 and 100 inclusive. Modbus read operations are more efficient in bigger batches of contiguous registers, but different devices have different limits on the size of the batched reads. This setting can also be helpful when building a modbus register map for an uncharted device. In some modbus devices a single invalid register in a read range will fail the entire read operation. By setting `scan_batching` to `1` each register will be scanned individually. This will be very inefficient and should not be used in production as it will saturate the link with many read operations.
63-
51+
| Field name | Required | Default | Description |
52+
| ---------- | -------- | ------- | ----------- |
53+
| ip | Required | N/A | The IP address of the modbus device to be polled. Presently only modbus TCP/IP is supported. |
54+
| port | Optional | 502 | The port on the modbus device to connect to. |
55+
| update_rate | Optional | 5 | The number of seconds between polls of the modbus device. |
56+
| address_offset | Optional | 0 | This offset is applied to every register address to accommodate different Modbus addressing systems. In many Modbus devices the first register is enumerated as 1, other times 0. See section 4.4 of the Modbus spec. |
57+
| variant | Optional | N/A | Allows variants of the ModbusTcpClient library to be used. Setting this to 'sungrow' enables support of SungrowModbusTcpClient. This library transparently decrypts the modbus comms with sungrow SH inverters running newer firmware versions. |
58+
| scan_batching | Optional | 100 | Must be between 1 and 100 inclusive. Modbus read operations are more efficient in bigger batches of contiguous registers, but different devices have different limits on the size of the batched reads. This setting can also be helpful when building a modbus register map for an uncharted device. In some modbus devices a single invalid register in a read range will fail the entire read operation. By setting `scan_batching` to `1` each register will be scanned individually. This will be very inefficient and should not be used in production as it will saturate the link with many read operations. |
59+
60+
### Register settings
6461
```yaml
6562
registers:
6663
- pub_topic: "forced_charge/mode"
@@ -83,24 +80,25 @@ registers:
8380
- pub_topic: "first_bit_of_second_byte"
8481
address: 13001
8582
mask: 0x0010
83+
- pub_topic: "load_control/optimized/end_time"
84+
address: 13013
85+
json_key: hours
86+
- pub_topic: "load_control/optimized/end_time"
87+
address: 13014
88+
json_key: minutes
8689
```
8790

8891
This section of the YAML lists all the modbus registers that you consider interesting.
8992

90-
`address` (Required) The decimal address of the register to read from the device, starting at 0. Many modbus devices enumerate registers beginning at 1, so beware.
91-
92-
`pub_topic` (Optional) This is the topic to which the value of this register will be published.
93-
94-
`set_topic` (Optional) Values published to this topic will be written to the Modbus device.
95-
96-
`retain` (Optional: default false) Controls whether the value of this register will be published with the retain bit set.
97-
98-
`pub_only_on_change` (Optional: default true) Controls whether this register will only be published if its value changed from the previous poll.
99-
100-
`table` (Optional: default 'holding') The Modbus table to read from the device. Must be 'holding' or 'input'.
101-
102-
`value_map` (Optional) A series of human-readable and raw values for the setting. This will be used to translate between human-readable values via MQTT to raw values via Modbus. If a value_map is set for a register the interface will reject raw values sent via MQTT. If value_map is not set the interface will try to set the Modbus register to that value. Note that the scale is applied after the value is read from Modbus and before it is written to Modbus.
103-
104-
`scale` (Optional: default 1) After reading a value from the Modbus register it will be multiplied by this scalar before being published to MQTT. Values published on this register's `set_topic` will be divided by this scalar before being written to Modbus.
105-
106-
`mask` (Optional: default 0xFFFF) This is a 16-bit number that can be used to select a part of a Modbus register to be referenced by this register. For example a mask of `0xFF00` will map to the most significant byte of the 16-bit Modbus register at `address`. A mask of `0x0001` will reference only the least significant bit of this register.
93+
| Field name | Required | Default | Description |
94+
| ---------- | -------- | ------- | ----------- |
95+
| address | Required | N/A | The decimal address of the register to read from the device, starting at 0. Many modbus devices enumerate registers beginning at 1, so beware. |
96+
| pub_topic | Optional | N/A | This is the topic to which the value of this register will be published. |
97+
| set_topic | Optional | N/A | Values published to this topic will be written to the Modbus device. Cannot yet be combined with json_key. See https://github.com/tjhowse/modbus4mqtt/issues/23 for details. |
98+
| retain | Optional | false | Controls whether the value of this register will be published with the retain bit set. |
99+
| pub_only_on_change | Optional | true | Controls whether this register will only be published if its value changed from the previous poll. |
100+
| table | Optional | holding | The Modbus table to read from the device. Must be 'holding' or 'input'. |
101+
| value_map | Optional | N/A | A series of human-readable and raw values for the setting. This will be used to translate between human-readable values via MQTT to raw values via Modbus. If a value_map is set for a register the interface will reject raw values sent via MQTT. If value_map is not set the interface will try to set the Modbus register to that value. Note that the scale is applied after the value is read from Modbus and before it is written to Modbus. |
102+
| scale | Optional | 1 | After reading a value from the Modbus register it will be multiplied by this scalar before being published to MQTT. Values published on this register's `set_topic` will be divided by this scalar before being written to Modbus. |
103+
| mask | Optional | 0xFFFF | This is a 16-bit number that can be used to select a part of a Modbus register to be referenced by this register. For example a mask of `0xFF00` will map to the most significant byte of the 16-bit Modbus register at `address`. A mask of `0x0001` will reference only the least significant bit of this register. |
104+
| json_key | Optional | N/A | The value of this register will be published to its pub_topic in JSON format. E.G. `{ key: value }` Registers with a json_key specified can share a pub_topic. All registers with shared pub_topics must have a json_key specified. In this way, multiple registers can be published to the same topic in a single JSON message. If any of the registers that share a pub_topic have the retain field set that will affect the published JSON message. Conflicting retain settings are invalid. The keys will be alphabetically sorted. |

‎modbus4mqtt/modbus4mqtt.py‎

Lines changed: 61 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
#!/usr/bin/python3
22

33
from time import sleep
4+
import json
45
import logging
56
from ruamel.yaml import YAML
67
import click
@@ -77,6 +78,10 @@ def poll(self):
7778
self.connect_modbus()
7879
return
7980

81+
# This is used to store values that are published as JSON messages rather than individual values
82+
json_messages = {}
83+
json_messages_retain = {}
84+
8085
for register in self._get_registers_with('pub_topic'):
8186
try:
8287
value = self._mb.get_value(register.get('table', 'holding'), register['address'])
@@ -98,8 +103,22 @@ def poll(self):
98103
if value in register['value_map'].values():
99104
# This is a bit weird...
100105
value = [human for human, raw in register['value_map'].items() if raw == value][0]
101-
retain = register.get('retain', False)
102-
self._mqtt_client.publish(self.prefix+register['pub_topic'], value, retain=retain)
106+
if register.get('json_key', False):
107+
# This value won't get published to MQTT immediately. It gets stored and sent at the end of the poll.
108+
if register['pub_topic'] not in json_messages:
109+
json_messages[register['pub_topic']] = {}
110+
json_messages_retain[register['pub_topic']] = False
111+
json_messages[register['pub_topic']][register['json_key']] = value
112+
if 'retain' in register:
113+
json_messages_retain[register['pub_topic']] = register['retain']
114+
else:
115+
retain = register.get('retain', False)
116+
self._mqtt_client.publish(self.prefix+register['pub_topic'], value, retain=retain)
117+
118+
# Transmit the queued JSON messages.
119+
for topic, message in json_messages.items():
120+
m = json.dumps(message, sort_keys=True)
121+
self._mqtt_client.publish(self.prefix+topic, m, retain=json_messages_retain[topic])
103122

104123
def _on_connect(self, client, userdata, flags, rc):
105124
if rc == 0:
@@ -121,6 +140,7 @@ def _on_subscribe(self, client, userdata, mid, granted_qos):
121140

122141
def _on_message(self, client, userdata, msg):
123142
# print("got a message: {}: {}".format(msg.topic, msg.payload))
143+
# TODO Handle json_key writes. https://github.com/tjhowse/modbus4mqtt/issues/23
124144
topic = msg.topic[len(self.prefix):]
125145
for register in [register for register in self.registers if 'set_topic' in register]:
126146
if topic != register['set_topic']:
@@ -147,9 +167,47 @@ def _on_message(self, client, userdata, msg):
147167
continue
148168
self._mb.set_value(register.get('table', 'holding'), register['address'], int(value), register.get('mask', 0xFFFF))
149169

170+
# This throws ValueError exceptions if the imported registers are invalid
171+
@staticmethod
172+
def _validate_registers(registers):
173+
all_pub_topics = set()
174+
duplicate_pub_topics = set()
175+
# Key: shared pub_topics, value: list of json_keys
176+
duplicate_json_keys = {}
177+
# Key: shared pub_topics, value: set of retain values (true/false)
178+
retain_setting = {}
179+
180+
# Look for duplicate pub_topics
181+
for register in registers:
182+
if register['pub_topic'] in all_pub_topics:
183+
duplicate_pub_topics.add(register['pub_topic'])
184+
duplicate_json_keys[register['pub_topic']] = []
185+
retain_setting[register['pub_topic']] = set()
186+
if 'json_key' in register and 'set_topic' in register:
187+
raise ValueError("Bad YAML configuration. Register with set_topic '{}' has a json_key specified. This is invalid. See https://github.com/tjhowse/modbus4mqtt/issues/23 for details.".format(register['set_topic']))
188+
all_pub_topics.add(register['pub_topic'])
189+
190+
# Check that all registers with duplicate pub topics have json_keys
191+
for register in registers:
192+
if register['pub_topic'] in duplicate_pub_topics:
193+
if 'json_key' not in register:
194+
raise ValueError("Bad YAML configuration. pub_topic '{}' duplicated across registers without json_key field. Registers that share a pub_topic must also have a unique json_key.".format(register['pub_topic']))
195+
if register['json_key'] in duplicate_json_keys[register['pub_topic']]:
196+
raise ValueError("Bad YAML configuration. pub_topic '{}' duplicated across registers with a duplicated json_key field. Registers that share a pub_topic must also have a unique json_key.".format(register['pub_topic']))
197+
duplicate_json_keys[register['pub_topic']] += [register['json_key']]
198+
if 'retain' in register:
199+
retain_setting[register['pub_topic']].add(register['retain'])
200+
# Check that there are no disagreements as to whether this pub_topic should be retained or not.
201+
for topic, retain_set in retain_setting.items():
202+
if len(retain_set) > 1:
203+
raise ValueError("Bad YAML configuration. pub_topic '{}' has conflicting retain settings.".format(topic))
204+
150205
def _load_modbus_config(self, path):
151206
yaml=YAML(typ='safe')
152-
return yaml.load(open(path,'r').read())
207+
result = yaml.load(open(path,'r').read())
208+
registers = [register for register in result['registers'] if 'pub_topic' in register]
209+
mqtt_interface._validate_registers(registers)
210+
return result
153211

154212
def loop_forever(self):
155213
while True:

‎modbus4mqtt/version.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
version="0.3.5"
1+
version="0.4.0"

‎tests/test_json_key.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
ip: 192.168.1.90
2+
registers:
3+
- pub_topic: "publish"
4+
json_key: "A"
5+
address: 1
6+
retain: true
7+
- pub_topic: "publish"
8+
address: 2
9+
json_key: "B"
10+
value_map:
11+
on: 1
12+
off: 2
13+
- pub_topic: "publish2"
14+
address: 3
15+
json_key: "A"

‎tests/test_mqtt.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,5 +366,73 @@ def test_address_offset(self):
366366

367367
mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish', 2, retain=False)
368368

369+
def test_json_key(self):
370+
# Validating the various json_key rules is among the responsibilities of test_register_validation() below.
371+
with patch('paho.mqtt.client.Client') as mock_mqtt:
372+
with patch('modbus4mqtt.modbus_interface.modbus_interface') as mock_modbus:
373+
mock_modbus().connect.side_effect = self.connect_success
374+
mock_modbus().get_value.side_effect = self.read_modbus_register
375+
376+
m = modbus4mqtt.mqtt_interface('kroopit', 1885, 'brengis', 'pranto', './tests/test_json_key.yaml', MQTT_TOPIC_PREFIX)
377+
m.connect()
378+
379+
self.modbus_tables['holding'][0] = 0
380+
self.modbus_tables['holding'][1] = 1
381+
self.modbus_tables['holding'][2] = 2
382+
self.modbus_tables['holding'][3] = 3
383+
m.poll()
384+
385+
mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish2', '{"A": 3}', retain=False)
386+
mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish', '{"A": 1, "B": "off"}', retain=True)
387+
388+
def test_register_validation(self):
389+
valids = [[ # Different json_keys for same topic
390+
{'address': 13049, 'json_key': 'a', 'pub_topic': 'ems/EMS_MODE'},
391+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'},
392+
{'address': 13050, 'json_key': 'b', 'pub_topic': 'ems/EMS_MODE'}
393+
],
394+
[ # Different topics, duplicate json_key
395+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEA'},
396+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEB'}
397+
],
398+
[ # Different topic, no json_key
399+
{'address': 13050, 'pub_topic': 'ems/EMS_MODEA'},
400+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEB'}
401+
],
402+
[ # Retain specified twice and consistent
403+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'retain': True},
404+
{'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': True}
405+
]]
406+
invalids = [[ # Duplicate json_key for a topic
407+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'},
408+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'}
409+
],
410+
[ # Missing json_key for a register with a duplicated pub_topic
411+
{'address': 13049, 'pub_topic': 'ems/EMS_MODE'},
412+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'}
413+
],
414+
[ # Retain specified twice and inconsistent
415+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'retain': True},
416+
{'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': False}
417+
],
418+
[ # set_topic and json_key both specified
419+
{'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'set_topic': 'ems/EMS_MODE/set', 'retain': True},
420+
{'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': False}
421+
]]
422+
for valid in valids:
423+
try:
424+
modbus4mqtt.mqtt_interface._validate_registers(valid)
425+
except:
426+
self.fail("Threw an exception checking a valid register configuration")
427+
for invalid in invalids:
428+
fail = False
429+
try:
430+
modbus4mqtt.mqtt_interface._validate_registers(invalid)
431+
except:
432+
fail = True
433+
if not fail:
434+
self.fail("Didn't throw an exception checking an invalid register configuration")
435+
436+
369437
if __name__ == "__main__":
370438
unittest.main()

0 commit comments

Comments
 (0)