core: support CSI/MIPI cameras (e.g IMX455, IMX477) on RPI CM4/CM5

[rmackay9]

Current behaviour

For my aerial photography quadcopter project I have an XFRobot C-20T camera gimbal with an RPI compatible camera with an IMX455 or IMX477 sensor connected to an RPI CM4/CM5 via CSI/MIPI. Could BlueOS be extended/fixed to support these sensors?

This BlueOS wiki page suggests that the Raspberry Pi Camera Module v2 can be used but I think support has actually be dropped at some point. I think BlueOS currently only supports USB and Ethernet cameras. Recent versions of Raspberry Pi OS support both the IMX455 and IMX477 sensors.

As a test I did manage to get the camera streaming on an RPI CM5 running BlueOS 1.5.0-beta26 but it was difficult:

1. update the kernel with, "sudo apt full-upgrade" so that "/boot/overlays/imx415.dtbo" was available
2. add some lines to /boot/firmware/config.txt

          dtparam=i2c_arm=on
          camera_auto_detect=0    <-- not sure this is actually necessary
          dtoverlay=imx415,cam0,addr=0x1a,4lane=1,clk-37125

3. install MediaMTX extension and adjust config:

          paths:
            cam:
              source: publisher
          rtspAddress: :8555

4. ssh onto the RPI CM5 (e.g Putty to pi@192.168.42.1) and run this command to publish to MediaMTX

          rpicam-vid -t 0 --width 1280 --height 720 --inline --codec yuv420 -o - | \
          docker exec -i extension-williangalvaniblueosextensionmediamtxv1114 sh -c \
          'ffmpeg -f rawvideo -pixel_format yuv420p -video_size 1280x720 -framerate 30 -i - \
          -c:v libx264 -preset ultrafast -tune zerolatency -f rtsp rtsp://127.0.0.1:8555/cam'

5. on my PC's VLC, Media, Open Network Stream, rtsp://192.168.42.1:8555/cam

I'm happy to provide camera(s) to help with the development and testing. Alternatively ArduCam has a wide range of cameras using the IMX477 sensor including this one.

Why is this important?

Giving the RPI direct control of the camera provides these benefits

1. improved control of the camera including resolution, frame rate and thus bandwidth required for live-video. The ethernet camera gimbals we've used so far (e.g. Siyi A8, XFRobot Z-1 mini) do not give us enough control over meaning we've needed to add a 2nd decoder/encoder to reduce the quality and bandwidth
2. reduced lag (decode/encoder mentioned above is skipped)
3. slightly reduced weight and heat (the camera gimbal is at least 10g lighter because it does need to process the sensor data itself, that is moved to the RPI's CPU)

Expected or desired behaviour

CSI/Mipi cameras using the IMX455 or IMX477 sensors would be usable with BlueOS meaning both Extensions and Video streaming to the GCS would work

Prerequisites

• I have checked to make sure that a similar request has not already been filed or fixed.

[github-actions]

Yo, this issue might be related to:

• core: camera-manager: add support for Raspberry Pi cameras #991: same CSI/Raspberry Pi camera support request with libcamera compatibility issues

[rmackay9]

I agree with Github that this is essentially the same issue as #991 although my issue has more detail. I'm happy to move my additions to the existing issue if people would prefer that.