ESP32C3 - GPIO7 fails to Output I2C SDA Signal when fs_mount() is invoked

[jumper385]

Hi All!

I have a custom ESP32C3 board that I want to use to log sensor data to a csv on an SD-Card. The sensors (MPU6050 and BME280) are connected to the I2C bus. The chip is talking to them via the sensor API's. I intend to use the Filesystem API to write logs.

Setup

GPIO6 - SCL
GPIO7 - SDA

Firmware Flow

The firmware works by:

1. Initialising I2C (&i2c0) and SPI (&spi2) threads. Additionally, each thread will have it's events initialised
2. Main thread will fire a mount event to SPI thread where fs_mount is invoked
3. Main thread will emit events to I2C thread to get IMU and Baro data
4. Unimplemented: Log data to CSV

Problem

Between when i remove step 2 (invoking fs_mount), step 3 can successfully run sensor_sample_fetch and subsequently sensor_sample_get() and sensor_value_to_float() from the sensors (wont throw an error). If i keep step 2, step 3 will not be able to successfully run sensor_sample_fetch (the return value is wrong). Additionally, the fs_mount wont throw an error and indicates that the SD-Card was mounted correctly.

Sadly, I'm not sure how to setup the ESP32 OpenOCD to step through firmware properly on zephyr to debug. It'd be great if someone could also point out how I could do this. If its easier, I'm happy to run it purely in terminal. west debug didn't seem to get me anywhere and information on setting up OpenOCD for ESP32 debugging is quite sparse. If needed, I'm also happy to write a brief document on how to set it up that could be used for the zephyr docs too!

Links

UWAA Flight Computer is the repo i'm currently working in. Its based off the example project.
Schematics: Note there are errors however, the pinout hasn't changed. The errors are 1) no pull-up on reset and 2) 32Mhz Clock NOT 40MHz clock. I've fixed these issues with a Jank and spare 40MHz clock.

[EricNRS]

The current Zephyr OpenOCD version 0.11 doesn't support ESP32 yet (not sure when v0.12 will be integrated), so you need to build the ESP32 OpenOCD build and then tell west to use that. See https://docs.zephyrproject.org/latest/boards/riscv/esp32c3_devkitm/doc/index.html#debugging for how to specify the OpenOCD version to use when doing west build which is then used when you do west debug.

To build the ESP32 Fork of OpenOCD, do the usual:

mkdir ~/esp32-openocd && cd $_
git clone --recursive https://github.com/espressif/openocd-esp32.git .
sudo apt-get install make libtool pkg-config autoconf automake texinfo libusb-1.0
./bootstrap  
./configure --prefix=$(pwd)/install  
make -j8 && make install

I set it up in this case to do the install to the source folder just to keep things together. You then follow this with the following commands from your Zephyr build directory:

export ESP_OPENOCDBASE=~/esp32-openocd/install
west build -b YOUR_BOARD YOUR_APP -- -DOPENOCD=${ESP_OPENOCDBASE}/bin/openocd -DOPENOCD_DEFAULT_PATH=${ESP_OPENOCDBASE}/share/openocd/scripts
west debug

As for your I2C and SD card issues it is hard to tell if you have a pin configuration issue or a threading issue. I would personally try to run the commands in a single thread as a test case to make sure that works. Also, check the final DT in build/zephyr/zephyr.dts to see if there is anything there that you are not expecting that would cause a pinctrl issue.

[EricNRS]

This is an open-source project and the Espressif team has limited time, so the best way to get help is to narrow down the issue and provide a patch to reproduce it with one of the sample apps. Often while narrowing it down, you find the root cause.

For debugging, once you get more than a few threads going, you will end up debugging a lot of things using logging. The Tracing framework is also useful, but I personally have not tried it yet (it is on my todo list).

Back to your issue, SPI and I2C should be unrelated, so another place to look is to see if you have an electrical issue with your board such that once you enable the SD card there is increased noise or a sag in the 3V3 power rail. Put an oscilloscope on the 3V3 bus and the I2C lines to see what is happening on the bus.

You can also enable logging for the I2C bus (I2C_LOG_LEVEL_DBG), the I2C sensor you are using, and the SPI driver for the flash chip and see what is happening. They may point you in the correct direction.

[rftafas]

Hi, thanks for your consideration.

They can also use Core Dump: https://blog.espressif.com/core-dump-a-powerful-tool-for-debugging-programs-in-zephyr-with-esp32-boards-969830fd6cdb

[jumper385]

Hi All!

Thanks for the input @rftafas! @EricNRS was correct - the firmware hasnt crashed so i didnt need a dump. I did start reading the logs of the device and found that the firmware was unable to transfer through I2C.

I probed the 3V3 Lines and found that no voltage sag was present - I even removed a protection diode and operated the line at 3V3 Volts. The same issue was the same.

I also probed the I2C lines and found something interesting. When fs_mount() wasnt invoked, both, the SDA and SCL were operating. When fs_mount() was invoked, however, only the SCL line was outputting - the SDA line was quiet.

Interesting aye...

[jumper385]

I may have found a solution but I'm not sure if this is something that should be flagged... Perhaps the docs may be enough to tell others and I may have been a bit dumb 😅

Findings

I probed the lines and found that the MTDO pin used for JTAG debugging was outputting a signal whenever the device would start up. When I switch the pins to a different pin (GPIO10), I am able to use both, SD Card and I2C. I'm not sure why this would stop the system from spitting out data from the I2C lines but i thought. Should this be flagged as a potential problem?

For reference, SDA was connected to GPIO7 and SCL was connected to GPIO6.

Waveforms

Multi-Threading Performance

EDIT: I've since tried this with the original firmware which had the SD-Card and I2C peripheral managed by separate threads. The current fix works.

[rftafas]

I understood this is a custom ESP32-C3 board, down to chip. First, make sure power and reset timings are correct. If they aren't, unexpected behavior may happen.

I did a sweep look into IDF reports and found some other issues, but all different from your report. You could try burning the e-fuses and permanently disable JTAG on pins (burning EFUSE_DIS_PAD_JTAG) to check if that still happens.

Additional:

• ESP32C3 TRM Chap. 4
• https://docs.espressif.com/projects/esp-idf/en/latest/esp32c3/api-reference/system/efuse.html
• https://github.com/espressif/esptool/releases/tag/v3.3

[EricNRS]

MTDO pin used for JTAG debugging was outputting a signal whenever the device would start up.

Nice work! Looks like you have the problem figured out and just need to nail down the root cause.

Check in the ESP32-C3 Technical Reference Manual - that is probably standard chip boot-up behaviour as it could also be a boot strapping pin or just be known to have a power-up glitch.

If either the SDA or SCL pins have glitchs on them, it is possible that your I2C devices end up in a bad state and require a reset to recover, so that would explain the issue you are seeing from a sensor standpoint.

[jumper385]

@rftafas Cheers for that! I can try to do that but not now - I'm running out of boards but will be doing a new revision of my hardware to test. i'll report back in a few weeks.
@EricNRS you are right - it's to do with the power-up glitch. I suspect @rftafas is right though, blowing the e-fuses would solve this permanently - specifically to eliminate the MTDO and MTCK glitches which are also seen in the plots (REF ESP32C3 Datasheet - Chapter 2.2 Table 2-2).

I do wonder why it does that when fs_mount is invoked. I'll do more digging.

[jumper385]

should i rename the discussion to JTAG Glitch Causes I2C line failure? Additionally, are there any other experiments you'd like me to run?

[EricNRS]

@rftafas Hi Ricardo. Henry isn't seeing a crash, he is just seeing that I2C sensor reads are failing when SD card (running in SPI mode) file system is successfully mounted. This is on an ESP32-C3.

[EricNRS]

On a similar topic of debug, do you know that state of Zephyr thread awareness in GDB? OpenOCD v0.12.0 added Zephyr thread awareness, but it doesn't seem to work yet (I have only tried on ESP32-S3 so far).

[rftafas]

Its out of topic, but for the sake of not leaving it hanging: it hasn't been tested yet, at least not by us, so I can't provide an status. Zephyr would first need to upgrade to 0.12 (AFAIK it is based on 0.11) and then we would start testing.

[EricNRS]

Thanks, I have opened an enhancement request #62791 to add Zephyr RTOS thread awareness to OpenOCD/GDB.

[jumper385]

why does #62791 not include the ESP32C3? is that to do with unknowns with the riscv core? I'm just curious at this point...

[EricNRS]

Mainly because I have only tested the support on ESP32-S3, although looking at the code, I don't see any support for RISC-V, either. I have updated the ticket to cover all ESP32-* processors.