Skip to content

Commit 6651fa6

Browse files
noursaidinoursaidi
authored
Documentation for running sequencer (#791)
1 parent 040eaaf commit 6651fa6

File tree

4 files changed

+167
-2
lines changed

4 files changed

+167
-2
lines changed

‎.wordlist.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,7 @@ unwriteable
162162
uplink
163163
url
164164
validator
165+
writeable
165166
writeback
166167
Writeback
167168
yaml

‎docs/guides/device_testing.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
[**UDMI**](../../) / [**Docs**](../) / [**Guides**](./) / [Device Testing with Sequencer](#)
2+
3+
# Device Testing with Sequencer
4+
5+
## Overview
6+
7+
The [sequencer](../tools/sequencer.md) tool runs automated testing against the device to determine compliance against parts of the UDMI specification
8+
9+
For complete device testing, sequencer can be used, however there are additional steps which need to be taken to ensure a successful test session with accurate results.
10+
11+
The device must support at a minimum the [basic state/config transaction](../specs/sequences/config.md). The device should be configured so that it is free of error, configured with at least a couple of points and be actively and frequently publishing data. The devices [`metadata.json`](../specs/site_model.md) file must contain the correct information for the device as configured, as well as include additional parameters which are exclusively used by sequencer during sequence testing. This is described in greater detail in the below headings.
12+
13+
## Requirements
14+
15+
#### Device's UDMI Implementation Prerequisites
16+
17+
The device must at a minimum support the [basic state/config transaction](../specs/sequences/config.md)
18+
19+
This comprises:
20+
21+
- Subscribe to the configuration topic (`/devices/{DEVICE_ID}/config`}
22+
- Upon receiving a **config** message, publish a **state** message to the state topic (`/devices/{DEVICE_ID/state`}, in which
23+
- The `system.last_config` property in the state message matches the `timestamp` property within the config message which was published
24+
25+
This is because the [sequencer](../tools/sequencer.md) tool will reset the configuration before each test runs, and ensure that the device is synchronized with the sequencer.
26+
27+
**IF** a device does not support this, [sequencer](../tools/sequencer.md) can still be used, however only a very limited subset of tests which do not require state messages can be run. This setting the `testing.noState` property to `true` in the devices metadata.json file, for example:
28+
29+
```json
30+
"testing": {
31+
"noState": true
32+
}
33+
```
34+
35+
#### Device Configuration Requirements
36+
37+
- The device must be online and actively publishing data
38+
- Recommended to set initial telemetry publish intervals to 20s so that the sequencer testing takes a reasonable amount of time and the correct results are produced.
39+
- The device must be fully and properly configured.
40+
- If the device is a gateway device, then a proxy device must be configured which must also meet all the requirements
41+
- The device must include at least two points, one of which is writeable
42+
- Under normal operating conditions, the device must not be reporting any status with a level of warning (400) or above in the state message
43+
- If there is, sequencer will fail all tests.
44+
45+
#### Sequencer/Metadata Configuration Requirements
46+
47+
The device's metadata.json file must accurately describe the device. In some tests, the sequencer will compare the data in metadata with the device that the data is publishing. Incorrect or missing data will result in the failure or skipping of tests.
48+
49+
- Points must match (`pointset.points`) what the device is publishing, and include any required properties to describe the device (e.g. `ref` fields)
50+
- The devices `system.hardware.make` and `system.hardware.model` must be correct and match what the device is publishing (e.g., if a device does include this property in its state message, and it reads `Google LLC`, the property must be the same. Just `Google` will prompt failure of these tests.
51+
- Similarly, the devices `system.software` must be a subset of the properties a device is publishing in its state message (if it does). If a device publishes multiple properties, e.g. `os`, `firmware`, and `build` - it is acceptable to only include one of these in the metadata, e.g. `firmware`.
52+
- The IPv4 (if applicable), IPv6 (if applicable) and MAC address are included in the localnet.families.ipv4, localnet.families.ipv4 and localnet.families.ether properties in the metadata
53+
54+
For example:
55+
56+
```json
57+
// rest of metadata file
58+
"system": {
59+
"hardware": {
60+
"make": "BOS",
61+
"model": "pubber"
62+
},
63+
"software": {
64+
"firmware": "v1"
65+
},
66+
},
67+
"localnet": {
68+
"families": {
69+
"ipv4": {
70+
"addr": "192.168.2.1"
71+
},
72+
"ether": {
73+
"addr": "00:50:b6:ed:5f:77"
74+
}
75+
}
76+
}
77+
```
78+
79+
Some tests use parameters defined in the metadata.json file. Presently, these are only for writeback testing. If a parameter is not defined, then the respective test is skipped. These are:
80+
81+
- `testing.targets.applied` - the `target_point` value is the name of a point which can successfully be written to, and the `target_value` is the value which can be successfully applied
82+
- `testing.targets.failure` - the `target_point` and `target_value` is the name of a point and the respective value which results in a `value_state` of `failure` when writeback is attempted
83+
- `testing.targets.invalid` - the `target_point` and `target_value` is the name of a point and the respective value which results in a `value_state` of `invalid` when writeback is attempted
84+
85+
For example,
86+
87+
```json
88+
// rest of metadata file
89+
"testing": {
90+
"targets": {
91+
"applied": {
92+
"target_point": "filter_differential_pressure_setpoint",
93+
"target_value": 60
94+
},
95+
"failure": {
96+
"target_point": "filter_alarm_pressure_status",
97+
"target_value": false
98+
},
99+
"invalid": {
100+
"target_point": "filter_differential_pressure_sensor",
101+
"target_value": 15
102+
}
103+
}
104+
```
105+
106+
For endpoint redirection testing, the `alt_registry` property in the `cloud_iot_config.json` file must be set and correspond to a valid registry in which the device is registered in. Run registrar within that registry (by changing the `registry_id` property temporarily) to ensure the device is registered.
107+
108+
## Testing Process
109+
110+
Before running the [sequencer](../tools/sequencer.md) for the first time, it is recommended to perform a few checks to ensure the device is correctly operating and ready for sequence testing. Otherwise, the sequencer will likely take a long time to run, and all tests will fail. This also mitigates against failures from site model errors, or stale site model files.
111+
112+
### 1. Run Registrar
113+
114+
Run [registrar](../tools/registrar.md) against the device to be tested to validate metadata files, and update the device configuration.
115+
116+
### 2. Run Validator
117+
118+
Run [validator](../tools/validator.md) and check that the:
119+
120+
- The device is functional online
121+
- The device is publishing data
122+
- The state message does not contain a status
123+
124+
### 3. Run Sequencer
125+
126+
[sequencer](../tools/sequencer.md) can be launched using the command:
127+
128+
```
129+
bin/sequencer PATH_TO_SITE_MODEL PROJECT_ID DEVICE_ID [SERIAL]
130+
```
131+
132+
[sequencer](../tools/sequencer.md) can take between ten minutes to an hour + to complete all the tests, depending on what functionalities a device supports.. This is because each test has a time limit. If a device supports a given functionality, it will pass the test, otherwise the sequencer waits until the time limit expires.
133+
134+
The [sequencer](../tools/sequencer.md) documentation provides additional run time options and settings.
135+
136+
When the sequencer is finished, a report is generated in the `out/devices/DEVICE_ID` directory. Complete test results, with logs, are in the `tests` sub-directory,
137+
138+
When tests fail, the report includes the failure reason, and an indication of where in the sequence the device failed.
139+
140+
## Tips and Common Failures
141+
142+
### Test failed: Timeout waiting for config sync
143+
144+
Synchronization between sequencer and the device has failed. This typically occurs when:
145+
146+
- The device is offline
147+
- The device does not support the state/config transaction, either:
148+
- Does not publish state
149+
- `last_config` does not match the `timestamp` of the last config sent
150+
151+
### Test failed: Timeout waiting for no applicable system status
152+
153+
This typically occurs when the device has a `status` in the `system` block of the state message. Sequencer requires that a device has no `status` before it commences testing. Refer to the details in the status message for additional information. This can typically be caused by:
154+
155+
- Device uses a template which includes a static status
156+
- The UDMI configuration is invalid for the device and should be correct
157+
- The local configuration of the device has something wrong in its configuration or another error

‎docs/guides/readme.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
# Guides
44

5+
- [Device Testing with Sequencer](device_testing.md) - A guide on how to use Sequencer for device testing
56
- [Development](development.md) - Information on contribution to the UDMI specification
67
- [Sequencing](sequencing.md) - Additional notes about (developing) sequence tests
78
- [Validator Troubleshooting](../tools/validator.md#troubleshooting) - Troubleshooting steps to help diagnose validator

‎docs/tools/sequencer.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,16 @@
11
[**UDMI**](../../) / [**Docs**](../) / [**Tools**](./) / [Sequencer](#)
22

3-
# Sequencer Setup
3+
# Sequencer
44

55
The UDMI _sequencer_ tool monitors a sequence of messages from a device's stream and
66
validates that the composition of sequential messages is compliant with the UDMI Schema
77

8+
# Complete Testing with Sequencer
9+
10+
A [guide is given for using sequencer for complete device testing.](../guides/device_testing.md)
11+
12+
# Sequencer Setup
13+
814
1. Ensure you have [deployed the necessary cloud functions](../cloud/gcp/udmis.md) to your GCP project
915
2. Add a new GCP IoT Core registry with a registry ID of `UDMI-REFLECT`.
1016
* Use `udmi_reflect` as the Pub/Sub topic for both the **device telemetry and **device state**
@@ -17,7 +23,7 @@ validates that the composition of sequential messages is compliant with the UDMI
1723
* device_id: Use the `<Registry ID>` as defined in Site Model for the devices to be tested.
1824
* auth_key: Use the public key you just created from `validator/rsa_public.pem`
1925

20-
## Running Sequencer
26+
# Running Sequencer
2127

2228
To run the _sequencer_, run the command from the top-level of the _site_model_ directory, with
2329
an optional device serial number:

0 commit comments

Comments
 (0)