video: add more resolutions to GC2145

[KurtE]

As I mentioned in issue #91968,

It would be great if the camera drivers would support a larger number of resolutions. For example those resolutions of the displays that they wish to display an image on. Potentially in both landscape and portrait mode.

So far I have added one the works in landscape mode for ILI948x and ST7796 displays as well

as a QVGA in portrait mode, which should be good for ILI9341 and some ST7789 displays.

If this looks like a valid approach will also add in GIGA Display Adapter size, and maybe and add the duplicate items to the FORMAT_CAP list for YUYV mode.

Note: the other than add the items to the list, I just added code to set resolution which computes the c_ratio/r_ratio chooses the smaller of the two to use.

Could probably do several of the others n the switch statement.

Example: shows GC2145 image shown on an ST7796 display on an Arduino GIGA using the Arduino wrapper.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[josuah]

This looks like some convenient "catch-all" rule to configure the exact resolution one needs but there is another way to do it:

• The role of video_set_format() is to configure the "effective resolution", the size after binning is appplied (the row and column ratio like you configure now).

• The role of video_set_selection() is to apply a cropping to the "effective/binned resolution" into an "output resolution" to give full control over what the sensor supports.

This should be merged this week so if it is possible for you to wait (or implement it!), then this would allow you to control this sensor features fully.

Thank you for the proposal!

[KurtE]

@josuah @dkalowsk @mjs513 and others...

As I mentioned when I first submitted this, I believe it would be a good thing to add some more commonly used
entries into the fmt table. Note: this is also likely true for several other of the cameras:
https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/video/gc2145.c#L779-L797
Currently it just has: 1600x1200, 640x480 and 320x240

And the set_resolution procesing only allows these three sizes:
https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/video/gc2145.c#L846-L877
And interestingly for these three sizes It chooses the ratio/scaling as:
size: 1600x1200 ratio: 1 Full size 1600x1200
size 640x480 Ratio: 2 full size 800x600 makes sense
size 320x240 ratio 3? full size = 533x400
Note: If you choose Ratio 4: full size 400 x 300
Or ration 5: full size 320x240:

Currently I am experimenting adding the functionality of video_get_selection/video_set_selection
#91878
Currently I have code in video_stm32_dcmi.c that forwards them to the sensor device:

static int video_stm32_dcmi_set_selection(const struct device *dev, struct video_selection *sel)
{
	const struct video_stm32_dcmi_config *config = dev->config;
	return video_set_selection(config->sensor_dev, sel);
}

static int video_stm32_dcmi_get_selection(const struct device *dev, struct video_selection *sel)
{
	const struct video_stm32_dcmi_config *config = dev->config;
	return video_get_selection(config->sensor_dev, sel);
}

I started with simple, just set the VIDEO_SEL_TGT_CROP in a test sketch to 480x320 and select 640x480 resolution and
I am getting the 480x320 output.

But if I try some of the updated video sample sketches, that use these messages, I see I need to handle some of the
additional selection types... Which I am experimenting with now.

For example suppose I wish to configure the GC2145 for something like 800 x 480 (GIGA Display Shield) And assuming
rotation such as if I can use Scale 2, It should allow from 800x600, how do I do this?
a) If I choose Resolution 640x480, that gives me that scaling factor, but I believe that within DCMIPP, that the
VIDEO_SEL_TGT_CROP code will see that the crop is bigger than the resolution. it will fail.

b) If I do get of VIDEO_SEL_TGT_NATIVE_SIZE and current resolution is set to 640x480, does it return
640x480 or 800x600?

c) if for example I have resolution: 640x480 and crop of 480x320, and set left and top to not be 0, I assume this
is allowing for me to pan around. What is left and top relative to? Specifically I am assuming that 640 within the
hardware width of 800 is offset by 80 (centered). I am assuming that by default crop of 480 would be default
set the actual crop to 160. Can I only pan +-80 within the 640x480 or can I pan the whole range of 0 to 800-480?

d)Looking at some of the example code, And then looking at DCMIPP.c I believe that After I set a TGT_CROP an dthen
set a selection, it may update the CROP board and I think maybe also sets TGT_COMPOSE?

e) Do I have additional control over the zoom level: That is suppose I wish for one of the resolution: 320x240
As I mentioned currently the code selects RATIO (scaling of 3)
But suppose I wish to control this, (i.e. camera zoom in or out). Scale of 5 would have me use the whole sensor for this,
4 would be: 400x300 with clipping and offset of 40x30... Maybe could go to scale 1 and just use a small portion of the
the sensor.

f) More of a side question: If the camera code supports the panning and zooming and so does the DCMIPP. Should DCMIPP (and other implementations) defer some of this to the cameras that support this? For example If I wish for for 480x320 where I have to choose 640x480, having the camera do it, would imply that it would clock out half as much data to the processor...
Just wondering.

Thanks in advance for insights here. I will continue to add in more of this support to the GC2145...
Thanks again

[josuah]

The ideal would be, to focus on the hardware capabilities: the format selection is expected to never change the field of view. For sensors that support cropping (most do), it is further possible to leverage the video_set_selection() API.

Then it becomes clear what to implement (defined according to the hardware), and how to use he APIs:

• First, configure a sensor resolution that is >= to the desired resolution.
• Then, use the video_set_selection() to reduce the resolution to the desired one.

The APIs are always called in this order: first the base resolution and then crop.
It makes the driver basic code that only maps the capabilities of the hardware.

a) If I choose Resolution 640x480, that gives me that scaling factor, but I believe that within DCMIPP, that the
VIDEO_SEL_TGT_CROP code will see that the crop is bigger than the resolution. it will fail.

I have to check but this sounds like harware-dependent. Some video hardware will allow adding padding anywhere around the video feed.

b) If I do get of VIDEO_SEL_TGT_NATIVE_SIZE and current resolution is set to 640x480, does it return 640x480 or 800x600?

=> Ideally, 800x600 (though could be 640x480 currently)

Maybe 640x480 should be removed from the driver list of formats altogether! :D
Not so shocking when considering that 640x480 is in fact not a native resolution of the GC2145 sensor, but instead:
1600x1200, 800x600 etc according to the binning level (aka "scaling factor").

To get 640x480, one must select a native size of 800x600 then crop. This is not about Zephyr APIs but about what the hardware needs to do, no way around it... So the APIs need to reflect what the hardware does.

c) if for example I have resolution: 640x480 and crop of 480x320, and set left and top to not be 0, I assume this
is allowing for me to pan around. What is left and top relative to? Specifically I am assuming that 640 within the
hardware width of 800 is offset by 80 (centered). I am assuming that by default crop of 480 would be default
set the actual crop to 160. Can I only pan +-80 within the 640x480 or can I pan the whole range of 0 to 800-480?

The rectangle is always absolute to the native format, and specifies the origin and the size:

struct video_rect {
        /** left offset of selection rectangle */
        uint32_t left;
        /** top offset of selection rectangle */
        uint32_t top;
        /** width of selection rectangle */
        uint32_t width;
        /** height of selection rectangle */
        uint32_t height;
};

If you wish to pan, change the origin to a different value, this is not recursive so should work.

d) Looking at some of the example code, And then looking at DCMIPP.c I believe that After I set a TGT_CROP an dthen
set a selection, it may update the CROP board and I think maybe also sets TGT_COMPOSE?

You may ignore "compose" (scale) in this case as not supported by DCMI, but this seems right...
Though "set a TGT_CROP" and "set a selection" and "update the CROP board" seems like the same thing.

also sets TGT_COMPOSE

This might make more sense when considering there is only one particular order users are expected to follow:

1. Set the format with video_set_format()
2. Set the cropping region with video_set_selection(dev, VIDEO_SET_TGT_CROP)
3. Set the scaling parameter with video_set_selection(dev, VIDEO_SET_TGT_COMPOSE)

Every step reset the values of what is below it: "select the native size, remove margins, and scale it up/down" always in this order.

e) Do I have additional control over the zoom level: That is suppose I wish for one of the resolution: 320x240
As I mentioned currently the code selects RATIO (scaling of 3) But suppose I wish to control this, (i.e. camera zoom in or out). Scale of 5 would have me use the whole sensor for this, 4 would be: 400x300 with clipping and offset of 40x30... Maybe could go to scale 1 and just use a small portion of the sensor.

"Zoom" is a very high-level feature that comes from the application.
One way to solve the question would be to wonder ourselves how we may do it with the hardware, forgetting about any API for now.

Zooming is cropping into an image, but up-scaling it to compensate the smaller output resolution.

In a DCMI-based hardware (i.e. no hardware scaler engine), how to perform zooming? The only 'scaling factors' we have are through software [wink] or through changing the sensor scaling factor.

Then we can crop this output to get the desired output size.

API-wise: select a different native resolution, then apply a different level of cropping.

Suddenly we realize the importance of not changing the field of view (cropping) at the level of the sensor video_set_format() as if we do, then we cannot reliably perform a zoom.

f) More of a side question: If the camera code supports the panning and zooming and so does the DCMIPP. Should DCMIPP (and other implementations) defer some of this to the cameras that support this? For example If I wish for for 480x320 where I have to choose 640x480, having the camera do it, would imply that it would clock out half as much data to the processor...

It is clever for an application to do this, but drivers are probably not a good place for clever decisions... and what if different drivers take different decisions... Best is to move all these decisions into the application, or eventually intermediate layers where these kind of "smarts" can be handled under the hood... Some needed work from Video area contributors!

The recording of this talk could interest you given your questions:
https://osseu2025.sched.com/event/25VpN/video4zephyr-from-basic-apis-towards-a-full-fledged-video-subsystem-phi-bang-nguyen-nxp-semiconductors?iframe=no

I hope this helped clarify a few points!

[josuah]

Note, it might look like the video APIs are a bit opaque, difficult to grasp. This might be due to the fact the documentation effort is standing behind the transformation of the APIs.

I believe there are 2 big transformations left:

• Replace the video format capability table by an enumerator, just like for the frame interval, to give more flexibility to the driver about how to store the formats (i.e. only store a list of resolution, and a separate list of pixel formats).

• Redo the buffering to be less inconvenient: switch to the RTIO library.

Then I suppose the video APIs will be more or less as they are expected to be, with minor changes, and documentation can start.

Thank you for your patience!

[KurtE]

This might make more sense when considering there is only one particular order users are expected to follow:

1. Set the format with video_set_format()
2. Set the cropping region with video_set_selection(dev, VIDEO_SET_TGT_CROP)
3. Set the scaling parameter with video_set_selection(dev, VIDEO_SET_TGT_COMPOSE)

Every step reset the values of what is below it: "select the native size, remove margins, and scale it up/down" always in this order.

Thanks @josuah,

That makes sense to me! However unless I am reading the code wrongly, the samples appear to have a complete different
order of things: For example the sample/drivers/video/capture (https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/video/capture/src/main.c#L151-L212): Which the code sort of goes like:

Note: in my test case I have set: CONFIG_VIDEO_FRAME_WIDTH=320 CONFIG_VIDEO_FRAME_HEIGHT=240
And no Crop, GC2145 is initially setup by default for 640x480

1. Set the crop if necessary - In this case crop not set
   /* Set the crop setting if necessary */

2. If frame WIDTH and HEIGHT are set: update fmt.. In my case fmt.width=320 fmt.height=240

3. Retrieve current crop: I returned ((0,0) 640x480 as this was the default format...

4. Compare CROP to FMT, Not same... So then it tries to call: VIDEO_SEL_TGT_COMPOSE which I currently do handle
   BUT if I did...
   Finally

5. It calls video_set_format

Which is completely opposite.... (I think)

[KurtE]

@josuah and .. I have had some luck with getting it up by defining crop the same as frame...

Currently as a test case I am using Nicla Vision, my own sample sketch which sends a captured image over USB.
My CDC ACM code is not great here, probably better stuff available.

If anyone wishes to play along: Currently the test code is up in:
https://github.com/KurtE/zephyr_test_sketches/tree/master/camera_capture_to_usb
I believe I have it self contained... Not using any of my other psuedo shared code.

Zephyr updated code is up at:
https://github.com/KurtE/zephyr/tree/camera_snapshot

Note: This fork/branch has an attempt at adding a snapshot mode to the video library.

At first I was trying to use the RAW image processing sketch that is part of the Arduino library, but it
has been problematic running on windows. So I have it setup to use the Arducam viewer. Example:

This was a capture with the Nicla Vision pointed at my IPAD which I was displaying a picture....
I am using the one that is in the extras of our teensy_camera library:
https://github.com/mjs513/Teensy_Camera

Probably will now experiment more with a snapshot mode, and with that hopefully allow the Nicla vision to capture
larger images, WIll also probably add overly/config for GIGA and/or Portenta.

[KurtE]

Replaced by #93797