Incorrect events decoding using numpy 2.x

[andberto]

Hi,
I'm trying to decode events from a ROS bag recorded with the sensor Prophesee EVK4 (IMX636) so the event codec should be EVT3.0.
Using the example provided I obtain events that are not correct (i.e x > senso_with, y > sensor_height, polarity not in {-1, 1} wrong timestamps).

Here's my code:

import argparse
import glob
import os
from bag_reader_ros2 import BagReader
from event_camera_py import Decoder

def extract_bag(input, output, event_topic):
    bag = BagReader(input, event_topic)
    decoder = Decoder()
    while bag.has_next():
        topic, msg, t_rec = bag.read_next()
        print(topic)
        decoder.decode_bytes(msg.encoding, msg.width, msg.height,
                             msg.time_base, msg.events.tobytes())
        cd_events = decoder.get_cd_events()
        print(cd_events)

def main(args=None):
    parser = argparse.ArgumentParser()
    parser.add_argument("path", help="ROS 2 bag file to extract or directory containing bags")
    parser.add_argument("--output_dir", default="/tmp/extracted_data", help="Folder where to extract the data")
    parser.add_argument("--events_topic", default="/event_camera/events", help="Event camera topic")
    args = parser.parse_args()

    bag_path = ''
    
    print(f'Data will be extracted in folder: {args.output_dir}')
    if not os.path.exists(args.output_dir):
        os.makedirs(args.output_dir)
    if os.path.isdir(args.path):
        bag_path = sorted(glob.glob(os.path.join(args.path, "*.db3")))[0]
    else:
        bag_path = args.path

    extract_bag(input = bag_path, output = args.output_dir, event_topic = args.events_topic)

if __name__ == '__main__':
    main()

Some events i'm obtaining:

[( 879, 16, 0, 5962275) (59904, 16948, 37, 14287178)
(64039, 90, 116, 1110764033) ... (32480, 16922, 100, -1017008928)
(24158, 52683, 1, 449) (32480, 16922, 100, -1017008928)]

Thank you in advance for the help.

[berndpfrommer]

Not much to go by from this.
The first question to settle: is this a problem with the python decoding, or is the data already bad.
To find out, please run an event camera renderer node and display the events when played back from a rosbag. Does the display of events look good?
Also, there is the event_ros_tools package that has a tool to dump the decoded data packets. Can you use that to decode? Do those packets also show the problem?
I see you are using decode_bytes(). What if you avoided it like this, would that work?

from bag_reader_ros2 import BagReader
from event_camera_py import Decoder

topic = '/event_camera/events'
bag = BagReader('foo', topic)
decoder = Decoder()

while bag.has_next():
    topic, msg, t_rec = bag.read_next()
    decoder.decode(msg)
    cd_events = decoder.get_cd_events()
    print(cd_events)
    trig_events = decoder.get_ext_trig_events()
    print(trig_events)

[andberto]

I've tried with the camera renderer and the events are displayed correctly, also with the dump tool from event_camera_tools the events are correct. I've tried your code snippet but the output is the same as mine. So the problem is with the python decoding i guess.

[berndpfrommer]

Would you mind sending me a ros2 bag that shows the problem? The shorter the better.

[andberto]

bag.zip
Here's a shortened version.

[berndpfrommer]

I ran the following code on your bag and it did not raise an exception. Strange. Could be a systems problem.

• what ROS2 version are you running, what OS, what hardware (x86_64 or arm64)?
• did you install the module from the apt repository? I used the latest "master" branch from github source.

from bag_reader_ros2 import BagReader

from event_camera_py import Decoder

topic = '/event_camera/events'
bag = BagReader('../bag', topic)
decoder = Decoder()

x_min, x_max, y_min, y_max = (0, 1280, 0, 720)
while bag.has_next():
    topic, msg, t_rec = bag.read_next()
    decoder.decode(msg)
    cd_events = decoder.get_cd_events()
    x = cd_events['x']
    y = cd_events['y']

    if (x < x_min).any() or (x >= x_max).any() or (y < y_min).any() or (y >= y_max).any():
        print(cd_events)
        raise Exception('event coordinates out of bounds')
print('no out-of-bounds coordinates found')

[andberto]

I'm using ROS Humble on ubuntu 22.04 (x86_64) i used the master branch from github (i followed the instruction on the repo). Maybe there is a problem with some python packages version.

[berndpfrommer]

Did you make any more progress on this issue?
Is installing the packages via apt an option?

[berndpfrommer]

Closing this due to inactivity.

[berndpfrommer]

OK, so I'm hitting the same issue when trying to use JAX. The decoder doesn't crash, but it gives garbled data. This has to do with the virtual environment I'm using. Looking into it.

[berndpfrommer]

The breakage occurs when I switch from numpy 1.x to numpy 2.x. The modules in this repo are compiled for numpy 1.x.
But how to fix this?

[berndpfrommer]

Apparently pybind11 updated the header with pybind 2.12 such that the modules would work with both Alas, even Ubuntu 24.04 currently only ships with pybind 2.11.1.

pybind/pybind11#5050

[berndpfrommer]

Solution to this problem:
Clone pybind11_vendor from here into your source workspace (along with the event_camera_py repo), then checkout the use_latest_pybind11 branch. Compile. This fixes the decoding errors for me.