[SOLVED]Need help interpreting the serial output of a sensor

[espruino-discuss3]

Posted at 2019-06-22 by larry

I bought a fine particle sensor, the sds011. This sensor has a USB socket that allows me to test it super fast with a Node.js script (SDS011-Wrapper) it works very well.

Now I want to make a "more" portable solution, by connecting the sensor with Espruino Pico.

This sensor is connected in Serial.

During my tests using a Node.js script, I have exactly an hex bytes buffer output that corresponds to the documentation, like :

<Buffer aa  c0  1b  05  7d  08  9c  eb  2c  ab>

For exemple, to get the PM2.5 value :
PM2.5 value: PM2.5 (μg /m3) = ((PM2.5 High byte *256) + PM2.5 low byte)/10

Where PM2.5 High byte = 05(Hex) = 5(decimal)
and PM2.5 low byte = 1b(Hex) = 27(decimal)

(reading de doc), after converting the Hex data to Decimal

const PM2.5 = ( ( 5 * 256 ) +   27   ) /10 = 130.7

Now, connecting Pico serial to this sensor and console.log the serial output give me weird result (exotic characters, like ÿ ô , etc ...). I know I missing something, I know I should convert the sensor serial output to decimal data but I don't know where to start?, well in fact I don't know what is the type of data the sensor give me? an array of Hex?, decimal ASCII values ?

Will apreciate your help if someone have already use this little cool device !
larry

[espruino-discuss3]

Posted at 2019-06-22 by Robin

Sat 2019.06.22

Hi @larry, garbled chars are usually a sign of baud rate mis-match.

http://www.espruino.com/Reference#l_Serial_setup

Would you please post results of process.env, the sample snippet of code that is performing the read, (best to present entire code block), and a sample output, so that we may see what is going on under the hood.

re: 'well in fact I don't know what is the type of data the sensor give me? '

In the #1 post, it is indicated that a buffer containing ten hex values was returned. Is this the case with the current Pico script?

[espruino-discuss3]

Posted at 2019-06-22 by larry

@robin

Here my small Node script, I've just connect de SDS011 to an USB to my computer (port COM8):

///////////////////////////////////
// Hello, SDS011
///////////////////////////////////
const SDS011Wrapper = require("sds011-wrapper");

const sensor = new SDS011Wrapper("COM8"); // Use your system path of SDS011 sensor.

console.log("process.env " , process.env);

sensor
    .setReportingMode('active')
    .then(() => {
        console.log("Sensor is now working in active mode.");
        return sensor.setWorkingPeriod(0); // Sensor will send data as soon as new data is available.
    })
    .then(() => {
        console.log("Working period set to 0 minutes.");
        console.log("\nSensor readings:");

        // Since working period was set to 0 and mode was set to active, this event will be emitted as soon as new data is received.
        sensor.on('measure', (data) => {
            console.log(`[${new Date().toISOString()}] ${JSON.stringify(data)}`);
        });
    });

I've added a console.log to know what the serialPort npm Node module serial output is like (I've edited the file wrapper.js in the node module directory : node_modules\sds011-wrapper\wrapper.js)

Here a sample view of wrapper.js where I've put my console.log:

const SerialPort = require('serialport');
const EventEmitter = require('events');

const SensorState = require("./core/sensor-state.js");
const SensorCommand = require("./core/sensor-command.js");

const addChecksumToCommandArray = require("./core/packet-utils.js").addChecksumToCommandArray;
const verifyPacket = require("./core/packet-utils.js").verifyPacket;

const PacketHandlers = require("./core/packet-handlers.js");

const ALLOWED_RETRIES = 10; // Number of retries allowed for single command request. 
const COMMAND_RETRY_INTERVAL = 150; // Time between sequential retries.

class SDS011Wrapper extends EventEmitter {

    /**
     * Open sensor.
     *
     * @param {string} portPath - Serial port path
     */
    constructor(portPath) {
        super();

        this._port = new SerialPort(portPath, { baudRate: 9600 });
        this._state = new SensorState();

        this._commandQueue = [];
        this._isCurrentlyProcessing = false;
        this._retryCount = 0;

        this._port.on('error', function (err) {
            console.log('Error: ', err.message);
        });

        this._port.on('close', () => {
            console.log('SDS011Wrapper port closed');
            this.close();
        });

        /**
          * Listen for incoming data and react: change internal state so queued commands know that they were completed or emit data.
          */
        this._port.on('data', (data) => {
            console.log("///// data ////// " , data);
            if (verifyPacket(data)) {
            ...
            ...

now the output :

///// data //////  <Buffer aa c0 f3 1b 1f 4e 9c eb 02 ab>
[2019-06-22T16:29:46.134Z] {"PM2.5":715.5,"PM10":1999.9}

you can see that the BaudRate is default to 9600 (wrapper.js), and I've set the same value when I use the sensor connected to Espruino Pico, here is my code :

function onInit() {
  
        Serial1.setup(9600, {rx: B7, tx: B6});
        Serial1.on('data', function (data) {
             console.log("> data  : " , data);
        }
}

with result like :

> data  :  ­
> data  :  ¶
> data  :  �
> data  :  «
> data  :  =
> data  :  ¼
> data  :  �
> data  :  ¶
> data  :  �
> data  :  ¯
> data  :  ¾
> data  :  ä
> data  :  q
> data  :  =
> data  :  }
> data  :  þ
> data  :  ú
> data  :  N
> data  :  �
> data  :  ë
> data  :  Ï
> data  :  ÿ
> data  :  �
> data  :  �
> data  :  N
> data  :  Ü
> data  :  ÿ
> data  :  ü
> data  :  ¡
> data  :  �
> data  :  �
> data  :  Î
> data  :  �
> data  :  ÿ

[espruino-discuss3]

Posted at 2019-06-22 by Robin

Ahhh, much better.

What version of Espruino is running? process.env

Now the fun begins!!

Everything appears okay at first glance. Have you confirmed Baud Rate in WebIDE? (guess yes)
Settings >> Communications >> Baud Rate

I had something like this a while back, and I was never entirely sure if the IDE needed the same Baud Rate setting.

Is it possible that Rx is not actually connected to Tx, and Tx -> Rx? e.g. pins reversed - worth a shot flipping them

Has a loopback test connecting the Pico Tx -> Rx been tried, in order to convince yourself the Pico works on the selected Serial Port?

Has a different Serial port been attempted? or, Software Serial?

Are those data packets returned only during the 'data' event? e.g. or is there other interval timing code that should be present?
Other things to try as was discovered (ref GND), different device though: > http://forum.espruino.com/conversations/334718/

[espruino-discuss3]

Posted at 2019-06-22 by AkosLukacs

Long shot (sorry, got to put the kids to sleep, so this is just my first guess): I think it's binary, that's why you see garbage characters.
Check out PMS7003's code for the buffer handling / decyphering. It might work similarly...

[espruino-discuss3]

Posted at 2019-06-23 by maze1980

You should replace

console.log("> data  : " , data);

with

for (index in data) { 
  console.log("> data:" + data.charCodeAt(index));
}

to see numbers instead of text.

[espruino-discuss3]

Posted at 2019-06-23 by Robin

Sun 2019.06.23

Has Serial.available() been attempted to see if a count of actual bytes are being received? This might/would prove out the baud issue.

http://www.espruino.com/Reference#t_l_Serial_available

Is there a clock line being used, and is it setup in both hardware and softwre correctly?

[espruino-discuss3]

Posted at 2019-06-24 by larry

first, thank you all for your post!

@robin

• Settings >> Communications >> Baud Rate set to 9600 > ok
• pins Tx/Rx reversed test > checked
• Same serial port (B7, B6 ) is working on other project of mine, tested on the same Pico (I will post the link of the complete tutorial on hackster.io probably tomorrow or before the end of this week), from now not yet tried to test with Software Serial on other Pico pin...
• console.log(Serial1.available()) give me 0 output, it seem's ok reading the doc.
• no clock line being used (in fact, I'm not sure what it is?, RTC module connected to the Pico?)
• process.env output :

"VERSION": "2v03",
"GIT_COMMIT": "e77d74f6",
"BOARD": "PICO_R1_3",
"FLASH": 393216, "RAM": 98304,
"SERIAL": "3f006300-04513432-33343134",
"CONSOLE": "USB",
"MODULES": "Flash,Storage,heatshrink,fs,net,dgram,tls,http,NetworkJS,WIZnet,tv,crypto,neopixel",
"EXPTR": 536871212

• Same Gnd for Pico and SDS011 : well, this thing, it can be a bit complicated:

Pico is connected by USB to my PC, and the SDS011 (5V needed) sensor is powered by an Breadboard Power Supply Module 3.3V 5V.
I do not think I can connect the Pico GND to the power supply module.
There may be another solution, I have a power boost module 3.3V to 5V. I could connect Pico pin 3.3V & Gnd output to the power boost module to power the SDS011, but would not I risk wanting too much power from the Pico and damaging it?

@AkosLukacs
Unfortunatly, I need more time and skill in the protocol Serie to adapt your code! but I will do some test on my project reading your scripts.

@maze1980

below, sample output of

    for (index in data) { 
      console.log("> data:" + data.charCodeAt(index));
    }

> data:170
> data:192
> data:134
> data:80
> data:239
> data:170
> data:192
> data:107
> data:179
> data:170
> data:192
> data:92
> data:251
> data:170
> data:192
> data:82
> data:255
> data:170
> data:192
> data:110
> data:170
> data:248
> data:251
> data:170
> data:252
> data:171
> data:255
> data:170
> data:255
> data:170
> data:255
> data:171
> data:255
> data:171
> data:253
> data:235
> data:247
> data:235
> data:253
> data:251
> data:245
> data:255
> data:245
> data:255
> data:171
> data:255
> data:183
> data:255
> data:107
> data:255
> data:91
...

well, It's maybe a correct output, comparing the output serial from the Node.js script when the sds011 module is running, I've got

    <Buffer aa  c0  1b  05  7d  08  9c  eb  2c  ab>

converting to decimal is

[170 192 27  5   125 8   156 235 44  171]

Reading the SDS011 doc, I should always get aa c0 (170 192 ) starting byte ... and ending with ab (171).
But, from my new decimal WebIDE output reading,
there not clean [170 192 data data data data data data data 171] repetitive sequence ?

I have to go to work now (already late)
thank's for your precious time folks!

[espruino-discuss3]

Posted at 2019-06-24 by AkosLukacs

Pico is connected by USB to my PC, and the SDS011 (5V needed) sensor is powered by an Breadboard Power Supply Module 3.3V 5V.
I do not think I can connect the Pico GND to the power supply module.

That (not connecting the two GNDs) is possibly the source of the "random" characters you see.
I guess you can get (almost) 5V from the Pico directly, but I don't have one, and wouldn't want to damage you stuff. So maybe someone with a Pico, or @gfwilliams can confirm, whether you can power the SDS11 from the **** pin :)

[gfwilliams]

Posted at 2019-06-24 by @gfwilliams

Sorry I'm a bit late to this - not connecting GND is likely to be a big cause of the mismatch of data you're getting. Connecting GND together on all devices won't cause you any issues at all (in fact not connecting GND is far more likely to break things as you can end up really stressing the IO pins).

The Pico has a 3.3v output which should be capable of about 200mA sustained, 400mA peak - however I'd only use that for stuff that needs regulated 3.3v (eg. not a power supply).

The Pico also has a VBAT pin which provides somewhere in the region of 4.5v when from USB or battery voltage when powered from battery, so I would use that: https://www.espruino.com/Pico#pinout

There's even a self-resetting fuse in there so if you draw too much power (>500mA sustained) from USB it'll just temporarily cut out without breaking anything.

It's entirely likely that the sensor will function perfectly fine from 4.5v without the boost converter - but if you're planning on switching to LiPo power and some point then you'd probably want the converter. When on LiPo you can just connect the LiPo power to BAT_IN and the Pico will automatically power from USB when USB is connected, or LiPo when USB isn't.

Just a note: The baud rate specified in the Web IDE will have no effect for you on the Pico, since the Pico is connected by USB.

[espruino-discuss3]

Posted at 2019-06-24 by larry

The Pico also has a VBAT pin which provides somewhere in the region of
4.5v when from USB or battery voltage when powered from battery, so I would use that: https://www.espruino.com/Pico#pinout

I'll try this tonight !

There's even a self-resetting fuse in there so if you draw too much
power (>500mA sustained) from USB it'll just temporarily cut out
without breaking anything.

didn't know that, it's cool

[espruino-discuss3]

Posted at 2019-06-24 by maze1980

Does the Pico request 500mA from the USB bus when connected to a PC? And what does it do when it's only allowed to use 100mA?

Anyway, assuming your USB port can provide 500mA that's enough for the Pico (32mA) and the sensor (70mA +/-10mA), no need for an external power supply.

When using multiple power supplies you MUST connect the ground of all power supplies together. Otherwise there might be a measurable voltage difference between the different grounds, e.g. due to power supply design or due to electrostatic charge.

-> Remove all data cables before connecting grounds together, otherwise you might damage your chips.

[espruino-discuss3]

Posted at 2019-06-24 by Robin

Mon 2019.06.24

@maze1980, as you have explained the ground issue correctly, am a bit puzzled by your first sentence then, even with the ground issue knowledge.

'Does the Pico request 500mA from the USB bus when connected to a PC? And what does it do when it's only allowed to use 100mA?'

Electronic devices connected to a supply (the load) will pull what ever is needed to meet the load demand. If the Pico is at idle, then it will only
draw the (32ma) as is indicated.

For the (100ma) situation, if there are no other parts connected to the regulator downstream, then the load draw as explained above applies. If say ten LEDs attempting to draw more than the (100ma) supplied, then those and the Pico will be limited to the (100ma) available, and the LEDs will be slightly dimmer and possibly the Pico may not function at all. A reason to always focus on the supply first, then design the circuit.

Remember this is Ohm Law: E = I * R   or   I = E / R   or   R = E / I

https://learn.sparkfun.com/tutorials/voltage-current-resistance-and-ohms-law/all

For USB ports, the slightly older USB2 can supply (500ma) as is indicated, while USB3 can supply up to (900ma)

https://www.extremetech.com/computing/115251-how-usb-charging-works-or-how-to-avoid-blowing-up-your-smartphone

[espruino-discuss3]

Posted at 2019-06-24 by maze1980

@robin: USB devices must be in low-power mode (max. 100 mA) initially. They can negotiate high-power mode (max. 500 mA for USB2.0), and only then they are allowed to use more power.
For the Pico itself it's not an issue.
But if you want connect additional devices to the Pico and power them by the USB port it might be an issue. While most PCs deliver 500mA per port some PCs and laptops don't. Connecting this sensor would draw slightly above 100mA, so it might damage the USB port (worst case scenario), that's why I asked.

[espruino-discuss3]

Posted at 2019-06-24 by larry

Powering SDS0111 module to Pico VBAT & GND give correct buffer serial sequence, how cool is that ? hum ? thank's bro, Espruino rock's, yeah!

got an PM2.5 sensor ready to go, I will be able to leave with and be disgusted to be completely in a cloud of pollution nearly all day long, cool!

> data:170
> data:192
> data:83
> data:8
> data:136
> data:12
> data:156
> data:235
> data:118
> data:171

> data:170
> data:192
> data:35
> data:8
> data:71
> data:12
> data:156
> data:235
> data:5
> data:171

> data:170
> data:192
> data:15
> data:8
> data:60
> data:12
> data:156
> data:235
> data:230
> data:171

> data:170
> data:192
> data:248
> data:7
> data:32
> data:12
> data:156
> data:235
> data:178
> data:171
...

[espruino-discuss3]

Posted at 2019-06-24 by Robin

@maze1980, thank you for reminding us on that limitation.

'so it might damage the USB port'

IMO I doubt it, unless one carelessly shorts the data pins or V+
The docs below indicate the ports limit the current unless enumeration is successful.

I have a three year old laptop with both USB2 and USB3 ports. It has always blindly allowed me to pull at least 500ma from the USB2 ports. Probably as it actually read this:

https://www.maximintegrated.com/en/app-notes/index.mvp/id/4803
bullet 1 "In USB 2.0, it is not strictly legal to draw power without enumerating, although much of present-day hardware does just that, and in violation of the spec."

> https://www.electronicdesign.com/interconnects/introduction-usb-power-delivery
> https://electronics.stackexchange.com/questions/5498/how-to-get-more-than-100ma-from-a-usb-port

The process of enumeration:

http://janaxelson.com/usbcenum.htm

[gfwilliams]

Posted at 2019-06-25 by @gfwilliams

@larry great! Glad you got it sorted!

@robin @maze1980 if you want to discuss the USB power delivery maybe we could do so in a new thread rather than sidetrack @larry's. But to confirm: Espruino does not request additional power from USB. I would be utterly staggered if any USB port could be damaged by drawing 500mA out of the power lines. Even if Espruino did request additional power it would be pointless because you're only supposed to draw the power after requesting it and there is no 'switched 5v' output that could be used.

[espruino-discuss3]

Posted at 2019-06-25 by maze1980

Regarding the USB power it's not worth implementing anything: I don't think any developer would ever check even if it was implemented.
@larry: Did your write a class module for this chipset?

[espruino-discuss3]

Posted at 2019-06-26 by larry

@maze1980 I'm on the way for it, I'll post a module to easily use the sensor here :) hope very soon !

[espruino-discuss3]

Posted at 2019-07-01 by larry

Here is the module for the SDS011, according to the manual to check the checksum and get proper value for PM2.5 and PM10.

const o = {};

function checkvalidity(bufferHex) {

    let  sum = 0;
    for (let i = 2; i < 8; i++) {
        sum = sum + parseInt(bufferHex[i], 16);
    }

    sum = sum &0xFF;
    const validCheckSum = parseInt(bufferHex[8],16);

    return sum == validCheckSum;
}

o.init = function (serial, callback) {
    console.log("> mod-dustsensor - init");

    //--- Exemple buffer
    //  data <Buffer aa  c0  1b  05  7d  08  9c  eb  2c  ab>
    //               170 192 27  5   125 8   156 235 44  171

    let bufferDecimal = [];
    let bufferHex = [];
    let bufferLength = 0;

    serial.on('data', function (data) {

        for (index in data) {
            let charDecimal = data.charCodeAt(index); // chope la valeur decimal du caractère
            let charHex = data.charCodeAt(index).toString(16); // chope la valeur Hex du caractère

            bufferDecimal.push(charDecimal);
            bufferHex.push(charHex);
            bufferLength++;

            if (+charDecimal == 171 && bufferLength == 10) {

                let validity = checkvalidity(bufferHex);

                if(validity){
                    // PM2.5 value -  ((PM2.5 High byte[3] *256) + PM2.5 low byte[2] ) / 10
                    // PM10 value -  ((PM10 High byte[5] *256) + PM10 low byte[4] ) / 10
                    let pm2_5 = ((bufferDecimal[3] * 256 ) + bufferDecimal[2])/10;
                    let pm10 = ((bufferDecimal[5] * 256 ) + bufferDecimal[4])/10;
                    callback({
                        pm2_5:pm2_5,
                        pm10:pm10
                    });
                }

                 bufferLength = 0;
                 bufferDecimal=[];
                 bufferHex=[];

            }
        }
    });
};

module.exports = o;

Entry point :

const modDustSensor = require('mod-sds011');

function onInit() {

    Serial1.setup(9600, {rx: B7, tx: B6});

    modDustSensor.init(Serial1, function(pmValues){
        console.log(`PM-2.5: ${pmValues.pm2_5} --- PM-10: ${pmValues.pm10} `);
    });

}

[espruino-discuss3]

Posted at 2019-07-01 by AkosLukacs

Nice!

If you get rid of bufferHex, and use normal numbers in checkValidity that should save some memory, and calls to parseInt.
Also, if you use a typed array: Uint8Array(10) as buffer, that should save some memory too.
And in checkvalidity you can use E.sum to sum the numbers (and Array.slice to get the data between the indexes). That should save some processor cycles, and code space as well.

Just replace this

function checkvalidity(bufferHex) {
    let  sum = 0;
    for (let i = 2; i < 8; i++) {
        sum = sum + parseInt(bufferHex[i], 16);
    }
    sum = sum &0xFF;
    const validCheckSum = parseInt(bufferHex[8],16);
    return sum == validCheckSum;
}

with

function checkvalidity(buff) {
    let sum =  E.sum(buff.slice(2,8));
    sum = sum & 0xFF;
    const validCheckSum = buff[8];
    return sum == validCheckSum;
}

I think I got the indexes right, tho just tested it in browser console with the sample in comments. :)

If you use Uint8Array, you have to use indexing to add data to the buffer, just replace

bufferDecimal.push(charDecimal);

with something like

bufferDecimal[bufferIndex++] = charDecimal;

And probably you have to change the buffer index checks to prevent overflow, if the data is not correct.
Sorry, I have a tendency to optimize code as I see... Hope you don't mind too much as effectively I have given you some work to do :)

[espruino-discuss3]

Posted at 2019-07-01 by larry

Hope you don't mind too much as effectively I have given you some work to do :)

yeah, it's a shame! nooo, it's the first time I try to interpret raw output data from sensor, so your comment helps me better understand on what's going on.

din't know about E.sum, i found more informations from the waveform exemple code very interresting, like :

Note: When dealing the large amounts of elements in Waveforms you'll start to notice Espruino's relatively slow execution speed. We'd recommend using E.sum, E.variance, E.convolve and E.FFT to work on large arrays wherever possible, and Array.forEach and Array.map to iterate over their elements.

I correct my duties at home, and I come back with a better copy, lol!

[espruino-discuss3]

Posted at 2019-07-01 by maze1980

            if (+charDecimal == 171 && bufferLength == 10) {

I'd call this highly optimistic, you would simply write

            if (bufferLength == 10) {

If you're out of sync for any reason there would be no more measurements, in both code variants.

[espruino-discuss3]

Posted at 2019-07-02 by larry

I did something more concise, following your advice:

const o = {};

function checkvalidity(buff) {
    let sum =  E.sum(buff.slice(2,8));
    sum = sum & 0xFF;
    const validCheckSum = buff[8];
    return sum == validCheckSum;
}

o.init = function (serial, callback) {

    //--- Exemple buffer
    //  data <Buffer aa  c0  1b  05  7d  08  9c  eb  2c  ab>
    //               170 192 27  5   125 8   156 235 44  171

    let bufferDecimal = [];
    let bufferLength = 0;

    serial.on('data', function (data) {

        for (index in data) {

            let charDecimal = data.charCodeAt(index); // chope la valeur decimal du caractère
            bufferDecimal.push(charDecimal);

            bufferLength++;

            if (bufferLength == 10) {


                let validity = checkvalidity(bufferDecimal);

                if(validity){
                    // PM2.5 value -  ((PM2.5 High byte[3] *256) + PM2.5 low byte[2] ) / 10
                    // PM10 value -  ((PM10 High byte[5] *256) + PM10 low byte[4] ) / 10
                    let pm2_5 = ((bufferDecimal[3] * 256 ) + bufferDecimal[2])/10;
                    let pm10 = ((bufferDecimal[5] * 256 ) + bufferDecimal[4])/10;

                    callback({
                        pm2_5:pm2_5,
                        pm10:pm10
                    });

                } else {
                    console.log("validity false");
                }

                bufferLength = 0;
                bufferDecimal=[];

            }
        }
    });
};

module.exports = o;

I 've started to build a closure with an soap plastic box, with inside the dust sensor, an ESP8266 or Espruino Pico and a small 128*64 Oled screen.

I would take data measurements while walking in Paris. I am curious about the results especially at the level of the big automobile hub, and close to the ring road.

I'll post the full project in the "Project" section of the forum if I do not encounter new problems ! (I have never used the Oled screen)

[espruino-discuss3]

Posted at 2019-07-02 by AkosLukacs

I'm working on something similar. OLED displays with Espruino are pretty easy, if you wire up it properly :)
https://luftdaten.info/ has a service & map where you can post dust-data (works outside of Germany too). They have a complete guide about how to make a stationary sensor with ESP8266, if you are interested.

[espruino-discuss3]

Posted at 2019-07-02 by larry

https://luftdaten.info/ has a service & map where you can post dust-data (works outside of Germany too). They have a complete guide about how to make a stationary sensor with ESP8266, if you are interested.

yes, I've seen this! maybe I'll build their module and contribute to!

[espruino-discuss3]

Posted at 2019-07-02 by @MaBecker

Just created an account, will try to get this working with Espruino firmware