Tip: Single command build, upload, run, show debug zen

[robertlipe]

In today's edition of "Stupid Shell Tricks"...

I've tuned my scripting environment to find a pretty happy place. With trivial scripting, it's possible to build mesmerizer, flash it to the board, wait for it to start running, and gobble up debugX output with no pointy clicky stuff and more importantly for me: no thinking. Just interrupt with ^C and command line repeat to do it all again because you're editing in another window in your favorite editor.

Start with a shell script that actually calls PIO to do the work. This is actually a one-liner in practice, but is mostly a collection of notes to myself:

$ cat go3
set -e

#~/.platformio/penv/bin/pio run  -e mesmerizer
~/.platformio/penv/bin/pio run --target upload -e mesmerizer
#~/.platformio/penv/bin/pio run --target buildfs -e mesmerizer
#[ -f /dev/cu.usbserial-110 ] && ~/.platformio/penv/bin/pio run --target uploadfs --upload-port /dev/cu.usbserial-110 -e mesmerizer
#[ -f /dev/cu.usbserial-101 ] && ~/.platformio/penv/bin/pio run --target uploadfs --upload-port /dev/cu.usbserial-101 -e mesmerizer

Then just run it. On successful completion, rely upon cat and stty both using stdin for input and just read the serial port:

➜ nightdriverstrip git:(balls) ✗ sh o3 && (stty 115200 ; cat ) < /dev/cu.usbserial-101

It'll keep reading the port and displaying it until you interrupt it. You can then repeat that command after you've made an edit. Of course, if there's a problem with either the build or the upload, the entire pipeline halts.

If you want to shave out a few more microseconds and get rid of the shell wrapper, you could eliminate that indirection away with something like:

~/.platformio/penv/bin/pio run --target upload -e mesmerizer && (stty 115200 ; cat ) < /dev/cu.usbserial-??1

Of course, you're still a slave to shipping a MB binary over a serial port even if you change only one byte in the image and with pio, a null build still takes about 15 seconds, and it uploads even if it's byte-identical, but I use this configuration because it's stateless. Whether a build succeeds or fails or there was an upload error or whatever, I can run that command (which we COULD compress into "go4"...) and just keep running it until the code is appropriately bending to your will.

Tips on how to make the remainder actually fast are welcome.

Oh, the serial ports on this SoC don't have serial numbers, so you have to chase physical locations. There's a way to get ioreg to divulge the logical mapping, or if you have only one mesmerizer attached, you know it'll be whichever /dev node matches /dev/cu.usbserial-???.

Enjoy!

(Is "Discussions" the right kind of place for this stuff? It feels wrong to not ask a question and wonder if it should be left open or closed.)

[davepl]

Cool! You might try upping the baud rate to 2000000, which works on many devices. If it works on mesmerizer, great! Should make the upload a little faster. You can also try OTA updates, which should work now, but they’re the same or slower. I think the flash write speed might be the gating factor. - Dave

…

On Jul 12, 2023, at 4:41 AM, Robert Lipe ***@***.***> wrote: In today's edition of "Stupid Shell Tricks"... I've tuned my scripting environment to find a pretty happy place. With trivial scripting, it's possible to build mesmerizer, flash it to the board, wait for it to start running, and gobble up debugX output with no pointy clicky stuff and more importantly for me: no thinking. Just interrupt with ^C and command line repeat to do it all again because you're editing in another window in your favorite editor. Start with a shell script that actually calls PIO to do the work. This is actually a one-liner in practice, but is mostly a collection of notes to myself: $ cat go3 set -e #~/.platformio/penv/bin/pio run -e mesmerizer ~/.platformio/penv/bin/pio run --target upload -e mesmerizer #~/.platformio/penv/bin/pio run --target buildfs -e mesmerizer #[ -f /dev/cu.usbserial-110 ] && ~/.platformio/penv/bin/pio run --target uploadfs --upload-port /dev/cu.usbserial-110 -e mesmerizer #[ -f /dev/cu.usbserial-101 ] && ~/.platformio/penv/bin/pio run --target uploadfs --upload-port /dev/cu.usbserial-101 -e mesmerizer Then just run it. On successful completion, rely upon cat and stty both using stdin for input and just read the serial port: ➜ nightdriverstrip git:(balls) ✗ sh o3 && (stty 115200 ; cat ) < /dev/cu.usbserial-101 It'll keep reading the port and displaying it until you interrupt it. You can then repeat that command after you've made an edit. Of course, if there's a problem with either the build or the upload, the entire pipeline halts. If you want to shave out a few more microseconds and get rid of the shell wrapper, you could eliminate that indirection away with something like: ~/.platformio/penv/bin/pio run --target upload -e mesmerizer && (stty 115200 ; cat ) < /dev/cu.usbserial-??1 Of course, you're still a slave to shipping a MB binary over a serial port even if you change only one byte in the image and with pio, a null build still takes about 15 seconds, and it uploads even if it's byte-identical, but I use this configuration because it's stateless. Whether a build succeeds or fails or there was an upload error or whatever, I can run that command (which we COULD compress into "go4"...) and just keep running it until the code is appropriately bending to your will. Tips on how to make the remainder actually fast are welcome. Oh, the serial ports on this SoC don't have serial numbers, so you have to chase physical locations. There's a way to get ioreg to divulge the logical mapping, or if you have only one mesmerizer attached, you know it'll be whichever /dev node matches /dev/cu.usbserial-???. Enjoy! (Is "Discussions" the right kind of place for this stuff? It feels wrong to not ask a question and wonder if it should be left open or closed.) — Reply to this email directly, view it on GitHub <#358>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4HCF36F73XJTE7TKYFBKTXP2EQNANCNFSM6AAAAAA2HLOPL4>. You are receiving this because you are subscribed to this thread.

[robertlipe]

2Mbps confirmed on mesmerizer on MacOS! (It's actually pretty hard to lose data in that direction over a USB port. There's just a ridic amount of buffering that happens behind your back.) It looks like the "do nothing" part of pio run is about 14 seconds and the actual uploading part is about 15. Spending an hour halving either one wouldn't have a huge change on workflow - not enough to buy back that hour.

For effect development, it's tempting to have a NightDriver core OS image that stays on the board and you just copy i the effects, treating it like a more traditional OS with a filesystem. That's more of an effort than I'm willing to put into saving that 30 seconds, even if I'm doing it 20x an hour.

[davepl]

My PC can do a full build from clean in 22 seconds, and my Mac can do it in 25, so at least the builds aren’t bad! Installable effects would be cool, but a major feature! The quickest way to develop effects is in C# on the PC and send the color data. You can deploy as fast as you can press F5, pretty much, but that only helps you if you’re doing remote effects! - Dave

…

On Jul 13, 2023, at 1:12 PM, Robert Lipe ***@***.***> wrote: 2Mbps confirmed on mesmerizer on MacOS! (It's actually pretty hard to lose data in that direction over a USB port. There's just a ridic amount of buffering that happens behind your back.) It looks like the "do nothing" part of pio run is about 14 seconds and the actual uploading part is about 15. Spending an hour halving either one wouldn't have a huge change on workflow - not enough to buy back that hour. For effect development, it's tempting to have a NightDriver core OS image that stays on the board and you just copy i the effects, treating it like a more traditional OS with a filesystem. That's more of an effort than I'm willing to put into saving that 30 seconds, even if I'm doing it 20x an hour. — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4HCF5B3EG5IRC6YN24W5DXQBJEJANCNFSM6AAAAAA2HLOPL4>. You are receiving this because you commented.

[robertlipe]

I think it's the psychological impact of knowing you changed 4 bytes in a .o and Wrote 1626784 bytes (945013 compressed) at 0x00010000 in 16.1 seconds (effective 807.1 kbit/s)... I suppose there might be an easier option: If you can't ship the bytes faster, just ship fewer of them. Create a minimizer target, strictly for effect dev, that disables web server, wait for DHCP, urlencode, json/spiffs, qr, jpeg decoder, lcd, and any other knobs that are easy to turn off. (I might leave audio on just because it's fun to integrate sometimes...) Is there an existing target that matches Mesmerizer's hardware pinout config, but is otherwise super minimal? Right now, it looks like Mesmerizer is kind of the showcase for all the features. I may experiment with that a bit this afternoon... It's really not something I plan to pursue too far, though a filesystem for effects IS an intriguing idea. With -S3, you have USB target mode, so it can pretend to be a FAT32 disk drive. Too many interesting project ideas in this code. Focus! On Thu, Jul 13, 2023 at 3:23 PM David W Plummer ***@***.***> wrote:

…

My PC can do a full build from clean in 22 seconds, and my Mac can do it in 25, so at least the builds aren’t bad! Installable effects would be cool, but a major feature! The quickest way to develop effects is in C# on the PC and send the color data. You can deploy as fast as you can press F5, pretty much, but that only helps you if you’re doing remote effects! - Dave > On Jul 13, 2023, at 1:12 PM, Robert Lipe ***@***.***> wrote: > > > 2Mbps confirmed on mesmerizer on MacOS! (It's actually pretty hard to lose data in that direction over a USB port. There's just a ridic amount of buffering that happens behind your back.) It looks like the "do nothing" part of pio run is about 14 seconds and the actual uploading part is about 15. Spending an hour halving either one wouldn't have a huge change on workflow - not enough to buy back that hour. > > For effect development, it's tempting to have a NightDriver core OS image that stays on the board and you just copy i the effects, treating it like a more traditional OS with a filesystem. That's more of an effort than I'm willing to put into saving that 30 seconds, even if I'm doing it 20x an hour. > > — > Reply to this email directly, view it on GitHub < #358 (comment)>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AA4HCF5B3EG5IRC6YN24W5DXQBJEJANCNFSM6AAAAAA2HLOPL4>. > You are receiving this because you commented. > — Reply to this email directly, view it on GitHub <#358 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD35F3D5XMW3R7VST353XQBKNNANCNFSM6AAAAAA2HLOPL4> . You are receiving this because you authored the thread.Message ID: <PlummersSoftwareLLC/NightDriverStrip/repo-discussions/358/comments/6443174 @github.com>

[robertlipe]

Wrote 1626784 bytes (945013 compressed) at 0x00010000 in 16.1 seconds (effective 807.1 kbit/s)...

I would never commit this, but with just some chainsaw surgery, disabling everything that makes mesmerizer cool: Wrote 1078976 bytes (597930 compressed) at 0x00010000 in 10.5 seconds (effective 820.0 kbit/s)... time ~/.platformio/penv/bin/pio run -e mesmerizer ========================= [SUCCESS] Took 54.95 seconds ========================= Environment Status Duration ------------- -------- ------------ mesmerizer SUCCESS 00:00:54.954 ========================= 1 succeeded in 00:00:54.954 ========================= ~/.platformio/penv/bin/pio run -e mesmerizer 258.81s user 56.86s system 571% cpu 55.247 total Is your M2 really twice as fast as my lowly M1 MBP?* I demand a recount!* Even in this build, though, I can tell it's compiling all kinds of things that are supposed to be configured away. I turned off REMOTE, but it's still building (and hopefully, throwing away as they should be unreferenced) objects like this: IRremoteESP8266/IRac.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/IRrecv.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/IRsend.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/IRtext.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/IRtimer.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/IRutils.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/ir_Airton.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/ir_Airwell.cpp.o Compiling .pio/build/mesmerizer/libbc4/IRremoteESP8266/ir_Aiwa.cpp.o There's FFT being built when there's no audio and such. Still, this is about as far down this rabbit hole as I'm going. I may clean up and add a 'minimizer' target, but More interesting problems to tackle... RJL

[robertlipe]

For others working on one board at a time, you can let presence of the devices's name be the hint to the target to build. Hoping it inspires others try something like this. (You'll have to sweeten to taste, of course.) I only have to reflash the filesystem when changing what I upload to the board, like a different project.

[ -c /dev/cu.usbserial-101 ] && {
  PORT=/dev/cu.usbserial-101
  BENV=mesmerizer
}
[ -c /dev/cu.wchusbserial556E0082011 ] && {
  PORT=/dev/cu.wchusbserial556E0082011
  BENV=demo
}

# PORT=/dev/cu.usbserial-101
# PORT=192.168.2.165

# esptool.py -p $PORT  erase_flash
# ~/.platformio/penv/bin/pio run -j6 --upload-port $PORT --target uploadfs -e $BENV
~/.platformio/penv/bin/pio run -j6 --upload-port $PORT --target upload -e  $BENV
(stty 115200 ; cat) < $PORT

Mesmerizer is just slow on the network update for me. The S3 is on part with your numbers, even working from the same tree. Even 'nc' times are all over with Mesmerizer.