Building a new version of MDBT42Q with different libraries.

[espruino-discuss]

Posted at 2019-03-03 by TomWS

I sadly discovered today that the build for MDBT42Q does not include FILESYSTEM. I looked at the MDBT42Q.py file and it appears that I can build a new version with FILESYSTEM and can also get rid of some stuff I don't need (GRAPHICS, NFC, NEOPIXEL). The thing I'm not certain about is whether I need the CRYPTO or AES set of libraries. Are these needed for some base security functions or can I dump these as well?

If I understand the process (which I am definitely NOT sure of), I simply create a new version of the file, changing the file name to something like MDBT42Q_tiny.py and build it... But which occurrences of 'MDBT42Q' do I change within the file and, on the make command, do I use BOARD=MDBT42Q_tiny?

Thanks in advance,
Tom

[espruino-discuss]

Posted at 2019-03-03 by Robin

Sun 2019.03.03

While there may be a design need to go the route described above, has the simpler means been reviewed?

http://www.espruino.com/Reference#l_E_connectSDCard
http://www.espruino.com/File+IO

---

Posted at 2019-03-04 by Wilberforce

You are in the right track. You don't need Aes or crypto.

What device do you intend the file system to be on? Are you going to wire up an sd card?

The file system has been ported on the esp32 to use flash memory - flashfs
If you have enough flash memory this could be an option. How big are your files?

---

Posted at 2019-03-04 by TomWS

My device uses the MDBT42Q module with a different set of IO devices than the Espruino MDBT42Q module. I don't have the LEDs or switch, but I do have: SD card, 6DOF accel/gyro, 3 axis magnetometer, and 64Mb Flash memory. The Espruino module is a good starting point, but, obviously, I need to change the libraries (NEOPIXEL? I don't think so...).
Thanks for the info on Crypto. Looks like everything will fit with it, but I might remove it in a later version when I start adding libraries for the other stuff. Right now I'm testing performance to see if this platform can handle my needs. I haven't had much success with OneWire, but that's tricky timing. Hopefully the rest of this will be ok...
I've built an image that I'll test tomorrow. Hopefully the learning curve is not too steep for my diminished capabilities ;-)

---

Posted at 2019-03-04 by @gfwilliams

What you're suggesting seems fine - copy the file, rename, tweak, build. It should be pretty easy

FILESYSTEM

Oops - my bad there. I've just added it to the build.

CRYPTO or AES set of libraries

They shouldn't be needed - the main one you might want is SHA1 websockets, but that's it.

NEOPIXEL? I don't think so...

Stuff like that doesn't take up much memory so it's not such a big deal to leave it in.

---

Posted at 2019-03-04 by TomWS

NEOPIXEL? I don't think so...
Stuff like that doesn't take up much memory so it's not such a big deal to leave it in.
I probably shouldn't have sounded so snobbish on NEOPIXEL, but, for me two things stand out with this device. Earlier ones, at least, the timing of the bit stream was very tight and long - something OneWire has proven difficult to do on my platform. The other thing is the power consumption of each device. It's not practical for a battery operated device, IMO.

Thanks for adding FILESYSTEM to the base. I'll test my 'tweak' today and use it for a learning experience, if nothing else...

---

Posted at 2019-03-04 by TomWS

Last night's build was successful and FILESYSTEM is now running on the MDBT42Q. Besides the libraries selections, I had to modify two things in the new BOARDNAME.py file:

1. I had to modify the binary name to distinguish it from the Official MDBT42Q binaries,
2. I modified BLUETOOTH_NAME_PREFIX to distinguish it as well (not sure if I needed to do this).

I did get a LOT of warnings about unable to 'inline' some functions. The build works, but it would be nice to know if this is 'normal'.

However, it's great to confirm that I can build a new image. I gives me an awesome sense of power! Bwahaha!

---

Posted at 2019-03-05 by @gfwilliams

I did get a LOT of warnings about unable to 'inline' some functions

Yep, it is - it's just a side-effect of link-time optimisation that Espruino uses (which allows some functions to be inlined across files).

---

Posted at 2019-04-12 by TomWS

I'm now testing multiple SPI ports on my version of the MDBT42Q board and have discovered a bit of 'chicken and egg' problem.

The Espruino build process uses BOARD.py to define the included features of a board. Ok, creating a new BOARD file and setting

'spi' : 3,

seemed to work and allowed me to pass a simple test:

I2C1.setup({scl:D10,sda:D11});
SPI2.setup({mosi:D07, miso:D05, sck:D04, baud:1000000});

function tellAbout(intf) {
  if(intf._options.mosi)  {
    console.log("interface is SPI");
    intf.cs = D09;
  } else {
    console.log("interface is I2C");
    intf.addr = 0x1c;
  }
  
  console.log(intf._options);
}

tellAbout(SPI2);
tellAbout(I2C1);

console.log(SPI2);
console.log(I2C1);

However, when I tried to actually USE the new port, the SPI hardware wasn't being used. I then looked at the jshardware.c file and saw that it was fixed at 'spi0' only, so I updated that to conditionally include code to instantiate and use spi1 & spi2. This, of course, required that I update sdk_config.h to set the respective SPIx_ENABLE macros (otherwise the nrf_drv...c files didn't know about these devices).

Once this was all done, the extra SPI ports worked marvelously.

Unfortunately, when I regressed the build for the original MDBT42Q I got a bunch of compiler errors stating that EVSPI2 & EVSPI3 were not defined. After a bit of research, I discovered the problem. EVSPIx are enumerated in jsdevices.h based on a macro SPI_COUNT defined in the auto-generated platform_config.h. Unfortunately, this macro is also defined in sdk_config.h based on the number of SPIx_ENABLE macros. As a result, using SPI_COUNT in jshardware.c file to conditionally build used the existing macro definition created in sdk_config.h due to the ordering of include files, overriding the definition in platform_config.h.

I suspected that reordering the include files in jshardware.c to try to solve this problem was probably the way to madness (DAMHIKT!) and ended up creating a new macro, SPI_MAX, in jsdevices.h, so that the proper board specific count of SPI devices could be saved and unambiguously used in jshardware.c.

This works for the nrf52 builds. SPI_MAX should be benign for all other boards. I think fundamentally having hardware definition controlled from two points (BOARD.py and the sdk_config.h files) is ripe for similar contentions. In normal nrf build structures the platform specific sdk_config.h is located in a platform specific config folder. However, this doesn't work well in this multi-platform situation so I'm not sure if there is an ideal solution. Maybe copying a BOARD specific version of sdk_config.h into the gen folder would work, but I'm not sure about the tooling to do this.

---

Posted at 2019-04-12 by @fanoush

So what happens when you try to use additional SPI devices and I2C together in js code (as these are sharing same hw resources and cannot be used concurrently)? Does the setup() fail for the SPI interface when it is already used by matching I2C and vice versa? There are in total 3 shared HW interfaces SPI0/TWI0, SPI1/TWI1 and SPI2.

---

Posted at 2019-04-12 by TomWS

@fanoush, I have a test case running where I use I2C and either SPI1 or SPI3. I did not try a test case where I try to use I2C and SPI2, although, if they weren't used concurrently, it might work, but, as you say, since they share HW resources, it's not a good idea. I don't think that there is any checking at the jswrapper level so it wouldn't be prohibited.

The jswrapper only enables TWI1, not TWI0.

---

Posted at 2019-04-15 by @gfwilliams

I think maybe the best solution would be to totally rename the SPI_COUNT/etc. IIRC they've already been renamed once to avoid a conflict with some other HAL, so should probably be prefixed JS_SPI_COUNT or something.

But as I'd said previously, increasing the hardware SPI count for nRF52 isn't something I'm that interested in doing unless we can handle the conflict with I2C in a way that alerts the user to a problem - especially as software SPI is really quite and you can have as many instances as you want.

If we're doing that, expanding I2C to allow I2C2 is probably an idea as well, but then you have the problem of what happens when someone tries to use I2C1 and SPI1 and it fails... Should it just silently use I2C2 instead?

---

Posted at 2019-04-15 by @fanoush

Well, the setup() method could simply fail with descriptive error (SPIx/TWIx already in use) if it is already setup by the other interface. So you would need to unsetup() one to use the other. Also the simplest and sensible to me is to let people handle it themselves and not hide which is which and silently use other.

Or if switching same interface between SPI/TWI is an issue at runtime then it could be at least static at build time how many of each one could use. So 0,1,2 of I2C in board config = up to 3,2,1 of SPI

As for usability - it would make better sense to use multiple HW SPIs if you could write to them asynchronously at the same time (like e.g. http). It could be useful e.g. for multiple displays. If the interpreter is waiting to write all bytes then indeed SW implementation might be enough if it is fast enough.

---

Posted at 2019-04-15 by TomWS

@gfwilliams, it would be pretty trivial to switch to a different macro, ie JS_SPI_COUNT, in the espruino files (4 in src folder, 4 in the 'targets/stm32' branch. Although, I'm not sure if stm32_it.c needs to be converted or not). The question is, where is platform_config.h generated as this would have to change as well.

@fanoush is correct, the I2C or SPI code could also verify if spi1_initialised or the i2c equivalent is set when trying to init the other and simply reject it. Multi-I2C would be useful where you have addressing conflicts or a mix of slow and fast devices.

@fanoush also brings up an relevant point WRT concurrency. I had a case where using EasyDMA was essential for performance on another project and I considered it in the 'too hard to do' on Espruino and stayed with the nordic tools instead. My current project doesn't require that performance, but, without that option, I'm not sure I'd take up that quest on future projects if I was the only one taking advantage of it. It's ironic that Espruino is probably more inclined to use the asynchronous interface with callbacks than other platforms.

---

Posted at 2019-04-15 by @gfwilliams

The SPI DMA support is definitely of interest - there's an issue open which includes it on espruino/Espruino#1212 (for all platforms).

One potential problem (why I didn't use it by default) is there's hardware bug which means that 1 byte SPI transfers fail, so you have to do some messing around to try and avoid it for those cases (and I think in earlier SDKs that might have meant modifying or avoiding the HAL altogether).

I could take a look at implementing the framework though (using the event queue to trigger a callback when the SPI transfer finished).

---

Posted at 2019-04-15 by TomWS

One potential problem (why I didn't use it by default) is there's hardware bug which means that 1 byte SPI transfers fail

There's another gotcha with nRF52 EasyDMA that I find annoying, the count on the SPI transfers is 8 bits which means that you can only transfer a maximum of 255 bytes, creating a really cumbersome sequence if you're transferring to/from something that has fixed 256 byte pages (like Flash memory to take a random example). You essentially have to transfer twice as many blocks for each page (transferring in 128 byte blocks, is the most efficient).

While concurrent SPI might be nirvana, there is a slight advantage to having muliple SPI ports if you're trying to lay out a tight board with multiple SPI devices. It's very easy to assign unique SPI pins to each device to simplify board routing. While the reconfigure-ability of the nRF52 pins doesn't REQUIRE you to use different SPI peripherals, using multiple peripherals does reduce the amount of reconfiguration necessary. I'll concede that this advantage, of course, is meaningless to an existing Espruino board, however.

---

Posted at 2019-04-16 by @gfwilliams

Thanks - that's a handy one to know as well. Do you know if those separate 255 byte transfers can be chained such that there isn't a pause in the SPI clock? I guess that might screw up the idea of using it as a general purpose neopixel driver.

As mentioned above though, without any concurrency from SPI you're not really going to see that much difference between using hardware SPI or the software SPI implementation though.

---

Posted at 2019-04-16 by TomWS

Do you know if those separate 255 byte transfers can be chained such that there isn't a pause in the SPI clock? I guess that might screw up the idea of using it as a general purpose neopixel driver.

It turns out I was somewhat incorrect on maximum number of bits you can transfer. There is a maximum of 255 bytes per SPI transaction, but, with EasyDMA operating in 'ArrayList' mode, the SPI transaction will be repeated (from the next contiguous address) without a 'pause'. Since the TX and RX is double buffered, presumably this means no pause in the SCK (although I haven't yet tried this). To STOP the transfer, you need to count the TX END events and issue a STOP to the peripheral. This could be done via PPI and a HW counter. This means that, if you can break the data block into 'n' equally sized blocks, then total transfer size is 'n' x TXD.MAXCNT.

I think the 'ArrayList' terminology is misleading and confusing. It implies (to me anyway) that you can have a list of transactions with independent and dis-contiguous buffers and sizes. Nordic's SPI Manager driver, however DOES provide this, but I doubt that the SPI clock is continuous between transactions. I've used the SPI Manager and it is powerful, but with power comes complexity.