Writing Sensor Data to File Leads to Delay Every 4kb (Arduino Nano Connect)

[fre-lax]

I'm reading the acceleration and gyroscope data from the LSM6DSOX-Chip on the Arduino Nano RP2040 Connect and store it in a file.
Since I am also storing a timestamp and another value, it is 6 axes + 2 → 8 values of 2 byte → 16 bytes. Those are saved as a binary file.

Unfortunately, every 4096 byte, there's a ~160ms delay. I think this is when some buffers are full/ emptied.
Probably it's the same problem as the person in this post had.

Is there a way to fix it? I want to save data with 100Hz, but having a delay every 2.5 Seconds screws up the data.

Here's the code (I don't know why i'ts not all in the code block..):

from machine import I2C
from machine import Pin
import ustruct
import utime
import random
`
def collect_data(time_s=10,file_name='file.csv',led=None,button=None):

### Init periphery and registers ###

# Initialize I2C
i2c = I2C(0, scl=Pin(13), sda=Pin(12), freq=400000)

# Sensor Address
SENSOR_ADDR = 0x6A  # or 0x6B

# Register Addresses
CTRL1_XL = 0x10  # Control register for accelerometer
CTRL2_G = 0x11  # Control register for gyroscope
OUTX_L_G = 0x22  # First register for gyro data output # page 70f 
OUTX_L_A = 0x28  # First register for accelerometer data output

SPI2_CTRL1_OIS = 0x70 # OIS configuration register

# Initialize accelerometer
i2c.writeto_mem(SENSOR_ADDR, CTRL1_XL, b'\x87')  # 1.66 kHz, 4g
# Initialize gyroscope
i2c.writeto_mem(SENSOR_ADDR, CTRL2_G, b'\x6C')  # 1.66 kHz, 2000 dps


def read_sensor():
    # Read 6 bytes of accelerometer data
    accel_data = i2c.readfrom_mem(SENSOR_ADDR, OUTX_L_A, 6)
    a = ustruct.unpack('<hhh', accel_data)

    # Read 6 bytes of gyroscope data
    gyro_data = i2c.readfrom_mem(SENSOR_ADDR, OUTX_L_G, 6)
    g = ustruct.unpack('<hhh', gyro_data)

    return a, g

### Measurement Intervals ###

frequency = 100 #Hz
period_ms = 1000/frequency

### LED Presets ###
if led:
    led_interval_s = 250/1000
    colors=[
        [255,0,0,0],
        [0,255,0,1],
        [0,0,255,2],
        [255,255,0,3],
        [255,0,255,4],
        #[0,255,255,5],
        #[255,255,255,6],
        [0,0,0,7]
    ]

    c=colors[1]
    led.update(color=c[:-1],brightness=10)

start_ms=utime.ticks_ms()

print(f'start_ms: {start_ms}')
t=0
last_change_t=t

with open(f'{file_name}.bin', 'wb') as binf:
    while t < time_s*1000 and button.value()==1:
        
        # get timestamp
        t=utime.ticks_diff(utime.ticks_ms(),start_ms)
        # to use less storage space, the time is saved as 2 bytes and the absolute time will be calculated when reading the file later
        t_save=t%(0xffff)

        # get sensor data
        a,g = read_sensor()

        # change led color randomly every set interval
        if led:
            if utime.ticks_diff(t,last_change_t)>led_interval_s*1000:
                last_change_t=t
                while True:
                    c_=random.choice(colors)
                    if c_!=c:
                        c=c_
                        break
                led.update(color=c[:-1])

        # saving measurements
        byte_data=bytearray(16)
        for i,reading in enumerate([a[0],a[1],a[2],g[0],g[1],g[2],t_save,c[-1]]):
            # Manually handle the conversion of the integer to 2 bytes
            byte_data[i*2] = reading & 0xFF
            byte_data[i*2+1] = (reading >> 8) & 0xFF
        # Write the byte data to the file
        binf.write(byte_data)

        # wait until next measuring point
        while abs(utime.ticks_diff(utime.ticks_diff(utime.ticks_ms(),start_ms),t))<period_ms:
            pass`

[andrewleech]

Unfortunately this kind of thing is normal with flash filesystems, flash chips are typically arranged into 4096 byte pages internally which need to be erased (slow) before write.

You may be able to restructure this to handle the sensor reads in a machine.Timer interrupt, storing into a buffer that the main loop reads to dump to file?
I'm not certain if the rp2 can still run code while its main flash is busy or not though.

You could possibly find a way to connect an SD card too and write to that instead.

[mendenm]

It might work to move the data collection to a thread on the second processor, and push the results to a queue. Then, have the main processor take chunks off the queue and write to a file. I suspect that the second processor won't be held up by the flash write from the first processor. The documentation of the XIP system indicates that it is designed to allow continued instruction reading while a block write is in progress. If you are very lucky, most of the code for the second processor will be sitting in the XIP cache, anyways.

[fre-lax]

EDIT:
See next post for the current version of the script

---

Thanks for the answer. I tried multithreading before, but could not get it to work.
I tried again now. It does work somehow, but not to the extend I was hoping:

1. If I take data with a higher frequency than 6 Hz, it will lead to an error: Exception occurred: [Errno 116] ETIMEDOUT

I think the reason for this is:
6 Hz -> 166 ms periods
7 Hz -> 142 ms periods
Writing a new datablock takes something about 160 ms
so when writing the data, it's trying to put a new value to the queue and that fails because the main thread is busy..

from machine import I2C
from machine import Pin
import ustruct
import utime
import random
import _thread

 ### Init periphery and registers ###

# Initialize I2C
i2c = I2C(0, scl=Pin(13), sda=Pin(12), freq=400000)

# Sensor Address
SENSOR_ADDR = 0x6A  # or 0x6B

# Register Addresses
CTRL1_XL = 0x10  # Control register for accelerometer
CTRL2_G = 0x11  # Control register for gyroscope
OUTX_L_G = 0x22  # First register for gyro data output # page 70f 
OUTX_L_A = 0x28  # First register for accelerometer data output

SPI2_CTRL1_OIS = 0x70 # OIS configuration register

readings_queue=[]
reading_active=False


def read_sensor():
    # Read 6 bytes of accelerometer data
    accel_data = i2c.readfrom_mem(SENSOR_ADDR, OUTX_L_A, 6)
    a = ustruct.unpack('<hhh', accel_data)

    # Read 6 bytes of gyroscope data
    gyro_data = i2c.readfrom_mem(SENSOR_ADDR, OUTX_L_G, 6)
    g = ustruct.unpack('<hhh', gyro_data)

    return a, g

def start_reading():
    print('thread started')
    global readings_queue
    global reading_active
    frequency = 5 #Hz
    period_ms = 1000/frequency
    start_ms=utime.ticks_ms()
    try:
        while True:
            if reading_active:
                t=utime.ticks_diff(utime.ticks_ms(),start_ms)
                # to use less storage space, the time is saved as 2 bytes and the absolute time will be calculated when reading the file later
                t_save=t%(0xffff)
                a,g = read_sensor()
                byte_data=bytearray(16)
                for i,reading in enumerate([a[0],a[1],a[2],g[0],g[1],g[2],t_save,t_save]): # add color for saving!
                    # Manually handle the conversion of the integer to 2 bytes
                    byte_data[i*2] = reading & 0xFF
                    byte_data[i*2+1] = (reading >> 8) & 0xFF
                # Write the byte data to the queue
                readings_queue.append(byte_data)
                if len(readings_queue)>500:
                    readings_queue=[]
                    print(f'{t_save}: queue getting too long')

                while abs(utime.ticks_diff(utime.ticks_diff(utime.ticks_ms(),start_ms),t))<period_ms:
                    pass
            else:
                 print('read it all')
                 return
    except Exception as e:
        print(f"Exception occurred: {e}")


def write_to_file(file):
            global readings_queue
            write_data = readings_queue
            readings_queue=[]
            for w in write_data:
                file.write(w)
            w_l=len(write_data)
            if w_l>0:
                print(f'written {w_l} lines:')
            
def collect_data(time_s=10,file_name='file.csv',led=None,button=None):
    global readings_queue
    global reading_active

    print(f'collecting data for {time_s} seconds...')

    # Initialize accelerometer
    i2c.writeto_mem(SENSOR_ADDR, CTRL1_XL, b'\x87')  # 1.66 kHz, 4g
    # Initialize gyroscope
    i2c.writeto_mem(SENSOR_ADDR, CTRL2_G, b'\x6C')  # 1.66 kHz, 2000 dps
   
    read_thread=_thread.start_new_thread(start_reading, ())

    reading_active=True

    with open(f'{file_name}.bin', 'wb') as binf:
        while button.value()==1:
            write_to_file(binf)
            utime.sleep_ms(3000)
    reading_active=False

[fre-lax]

First:

Could this frequency be a problem?

i2c = I2C(0, scl=Pin(13), sda=Pin(12), freq=400000)

I changed the whole code and created a class for reading the data.
The class object is initiated within the second thread and will push the data to a global variable every 50 cycles.

The data reading method is called by a timer interrupt (100Hz).

Whenever the global variable contains 100 or more lines, it will be written to the file.

Unfortunately, the writing process will still slow down the reading every 2 or 3 writings when surpassing another 256 lines = 4096 bytes

Interestingly, the timer interrupt will "catch up" and trigger multiple times in a row very fast. I saw this when I printed the cycle time of the data collection to the console:

Writing within a 4kb block:

10
10
10
push
writing 100 lines
10
10
10

Writing when new 4kb block is created (average cycle time is 12.xx ms for the cycles that are not exactly 10 ms) :

10
10
10
push
writing 100 lines
72
10
1
76
10
2
1
1
0
1
1
1
1
3
10
10
10
10

from machine import I2C
from machine import Pin
from machine import Timer
import utime
import _thread


readings=[]
stop_reading=False

class READ_SENSOR():
    # Sensor Address
    SENSOR_ADDR = 0x6A  # or 0x6B

    # Register Addresses
    CTRL1_XL = 0x10  # Control register for accelerometer
    CTRL2_G = 0x11  # Control register for gyroscope
    OUTX_L_G = 0x22  # First register for gyro data output # page 70f 
    OUTX_L_A = 0x28  # First register for accelerometer data output

    SPI2_CTRL1_OIS = 0x70 # OIS configuration register
     
    def __init__(self) -> None:
        print('init class')
        self.data=[]
        
        # Initialize I2C
        self.i2c = I2C(0, scl=Pin(13), sda=Pin(12), freq=400000)

        # Initialize accelerometer
        self.i2c.writeto_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.CTRL1_XL, b'\x87')  # 1.66 kHz, 4g
        # Initialize gyroscope
        self.i2c.writeto_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.CTRL2_G, b'\x6C')  # 1.66 kHz, 2000 dps
        
        self.frequency = 5 #Hz
        self.period_ms = 1000/self.frequency
        self.start_ms=utime.ticks_ms()
        
        #self.t0=self.start_ms

        print("class initialized")

    def read_sensor(self,t):

        byte_data=bytearray(16)
        t_now=utime.ticks_diff(utime.ticks_ms(),self.start_ms)

        # print(utime.ticks_diff(t_now,self.t0))
        # self.t0=t_now
        
        # to use less storage space, the time is saved as 2 bytes and the absolute time will be calculated when reading the file later
        t_save=t_now%(0xffff)

        byte_data[0:12]=self.i2c.readfrom_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.OUTX_L_A, 12)
        byte_data[12]=t_save & 0xFF
        byte_data[13]=(t_save >> 8)& 0xFF
        # byte_data[14]=
        # byte_data[15]=
        self.data.append(byte_data)

        if len(self.data)>=50:
            print('push')
            self.update_data(None)

    def update_data(self,t):
        global readings
        readings+=self.data
        self.data=[]

def start_collecting(frequency):
    print('start collecting')
    global readings

    Reader = READ_SENSOR()

    print(f'read every {int(1/frequency*1000)} ms')
    print('init timers')

    read_timer = Timer()
    read_timer.init(mode=Timer.PERIODIC, freq=frequency, callback=Reader.read_sensor)


def write_to_file(file):
    global readings
    write_data = readings
    readings=[]
    for w in write_data:
        file.write(w)

def collect_data(time_s=10,file_name='file.csv',led=None,button=None):
    print(f'collecting data: {file_name}')
    global readings
    global stop_reading
    f = 100 #Hz
    readings_thread = _thread.start_new_thread(start_collecting, (f,))
    with open(f'{file_name}.bin', 'wb') as binf:
        while button.value()==1:
            if len(readings)>=100:
                print(f'writing {len(readings)} lines')
                write_to_file(binf)
    stop_reading = True
        
    print('done')

[mendenm]

you may want to double-buffer the queue, that is, write to the queue as you do, but on the consumer end, pop data off the queue into a secondary buffer, and the write to disk from the secondary buffer. I would also use thread safe locks to lock and unlock the primary queue when you are writing and reading data from it. Since the read and write operations on the queue will be very fast, that should never take enough time to block your data producer.

[fre-lax]

Thank you again for the feedback.

It seems I can't get rid of the delays totally.

I now managed to stretch it out to a longer delay every 8 seconds which helps a little bit:

Here you can see the cycle time in ms (y-axis) in ms for each cycle (x-axis):

Here's the current code:

from machine import I2C
from machine import Pin
from machine import Timer
import utime
import _thread


readings=[]
doublebuffer=[]
stop_reading=False
lock = _thread.allocate_lock()

class READ_SENSOR():
    # Sensor Address
    SENSOR_ADDR = 0x6A  # or 0x6B

    # Register Addresses
    CTRL1_XL = 0x10  # Control register for accelerometer
    CTRL2_G = 0x11  # Control register for gyroscope
    OUTX_L_G = 0x22  # First register for gyro data output # page 70f 
    OUTX_L_A = 0x28  # First register for accelerometer data output

    SPI2_CTRL1_OIS = 0x70 # OIS configuration register
     
    def __init__(self) -> None:
        print('init class')
        self.data=[]
        
        # Initialize I2C
        self.i2c = I2C(0, scl=Pin(13), sda=Pin(12), freq=400000)

        # Initialize accelerometer
        self.i2c.writeto_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.CTRL1_XL, b'\x87')  # 1.66 kHz, 4g
        # Initialize gyroscope
        self.i2c.writeto_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.CTRL2_G, b'\x6C')  # 1.66 kHz, 2000 dps
        
        self.frequency = 5 #Hz
        self.period_ms = 1000/self.frequency
        self.start_ms=utime.ticks_ms()
        
        self.t0=self.start_ms

        print("class initialized")

    def read_sensor(self,t):

        byte_data=bytearray(16)
        t_now=utime.ticks_diff(utime.ticks_ms(),self.start_ms)
        dt=utime.ticks_diff(t_now,self.t0)
        if not (9<= int(dt) <= 11):
            print(dt)
        self.t0=t_now
        
        # to use less storage space, the time is saved as 2 bytes and the absolute time will be calculated when reading the file later
        t_save=t_now%(0xffff)

        byte_data[0:12]=self.i2c.readfrom_mem(READ_SENSOR.SENSOR_ADDR, READ_SENSOR.OUTX_L_G, 12)
        byte_data[12]=t_save & 0xFF
        byte_data[13]=(t_save >> 8)& 0xFF
        # byte_data[14]=
        # byte_data[15]=
        self.data.append(byte_data)
        
        if len(self.data)>=200:
            self.update_data(None)

    def update_data(self,t):
        with lock:
            global readings
            print('push:',end=' ')
            readings+=self.data
            self.data=[]
            print(len(readings))

def start_collecting(frequency):
    print('start collecting')

    Reader = READ_SENSOR()

    print(f'read every {int(1/frequency*1000)} ms')
    print('init timers')

    read_timer = Timer()
    read_timer.init(mode=Timer.PERIODIC, freq=frequency, callback=Reader.read_sensor)


def write_to_file(file):
    global doublebuffer
    write_data = doublebuffer
    doublebuffer=[]
    for w in write_data:
        file.write(w)

def push_double_buffer():
    with lock:
        global doublebuffer
        global readings
        doublebuffer+=readings
        readings=[]

def collect_data(time_s=10,file_name='file.csv',led=None,button=None):
    print(f'collecting data: {file_name}')
    global readings
    global stop_reading
    global doublebuffer
    f = 100 #Hz
    readings_thread = _thread.start_new_thread(start_collecting, (f,))
    with open(f'{file_name}.bin', 'wb') as binf:
        while button.value()==1:
            utime.sleep_ms(2)
            if len(readings)>=200:
                push_double_buffer()
            if len(doublebuffer)>=800:
                print(f'writing {len(doublebuffer)} lines')
                write_to_file(binf)

        stop_reading = True
        while button.value()==0:
            pass
        
    print('done')

[mendenm]

Try one more thing: put a gc.collect() in your main loop. You are allocating a lot of objects (the bytearray(16) sub-buffers) and I wonder if you have hit the gc time. doing it more regularly makes more, shorter bumps.

[fre-lax]

It does change something: If I put it here, it messes up everything:

while button.value()==1:
    gc.collect()
    utime.sleep_ms(2)

If I put it in the read_sensor() method, it increases the cycle time to 12-15 ms but seems to decrease the writing time as well...
Strangely theres sometimes randomly longer cycles of ~80ms.

And unfortunately it messes with my button which is not working anymore. So I can only stop the script by resetting the microcontroller. The binary file seems to be empty after this.