Introduce RGB/DPI driver

[Dominaezzz]

Thank you for your contribution!

We appreciate the time and effort you've put into this pull request.
To help us review it efficiently, please ensure you've gone through the following checklist:

Submission Checklist 📝

• I have updated existing examples or added new ones (if applicable).
• I have used cargo xtask fmt-packages command to ensure that all changed code is formatted correctly.
• My changes were added to the CHANGELOG.md in the proper section.
• I have added necessary changes to user code to the Migration Guide.
• My changes are in accordance to the esp-rs API guidelines

Extra:

• I have read the CONTRIBUTING.md guide and followed its instructions.

Pull Request Details 📖

Description

Add a driver for the RGB mode of the LCD_CAM peripheral. Closes #2081.
I've called it Dpi since Rgb is a generic name (though I don't mind renaming it if anyone insists). The I8080 driver has similar opinionated name.

I've opened this early as a formal request for one of the maintainers to acquire a devkit that can run the example.
Makerfabs MaTouch_ESP32-S3 Parallel TFT with Touch 4.3"

(I tried using the espressif devkit linked in the issue but the initialization code is buried deep in esp-bsp and it would complicate the example with bitbanging SPI over an I2C gpio expander)

The example runs on the official espressif devkit ESP32-S3-LCD-EV-Board.

Testing

HIL test and example

[Georges760]

Great to see this PR, I was considering working on this need.

[jgtaylor]

I've tested this on a 7" with an EK9716 display controller, and it appears to be working. Board used: https://www.aliexpress.com/item/1005005560920555.html

I've not done any kind of exhaustive tests - just run the lcd_dpi.rs from the example directories (with pins modified for the above board).

The example code results in the screen going through various colors. On my device, it appears to flicker fairly intensely. Changing the clock to 40 & 50 Mhz seems to help with the flicker a bit. Using 33 Mhz results in interesting artifacts on 2/3 of the display.

In any case, this MR has my vote! Getting this into main would be a tremendous step forward!

[Dominaezzz]

The example code results in the screen going through various colors. On my device, it appears to flicker fairly intensely. Changing the clock to 40 & 50 Mhz seems to help with the flicker a bit. Using 33 Mhz results in interesting artifacts on 2/3 of the display.

Curious, in addition to the pins, did you also update the frame timings?
Also maybe try making the loop buffer bugger.
I'd also appreciate a video recording of the flickering if you can share one.

[jgtaylor]

please forgive the horrible video and my silly commentary (also, apologies for using YouTube) - https://www.youtube.com/shorts/tHPwoWvb11o

Also, I had to turn on the backlight to actually see the screen. The flicker could be from the backlight, or not using ledc to dim it... or, or or ... I confess, I'm so new to rust, I'm still fighting with the borrow checker, but I am excited as can be to get this working!

[jgtaylor]

quick update - i've played with the timings, however they're pretty spot on for the EK9716. I increased the dma_loop_buf to 16 * 16 which helped. It removed the artifacts at 33.MHz(). I think my flicker is more a result of the back-light.

somewhat random question: do the "back porch" timings need to be defined? Pic from datasheet for posterity, or in-case it helps anything (pls ignore if its useless)

[Dominaezzz]

please forgive the horrible video

I suppose it can't be helped for this example since every single frame is a different color. Trying to see it on video was a lost cause to begin with haha. Would need a different example, one that alternates between two colors once a second.

somewhat random question: do the "back porch" timings need to be defined? Pic from datasheet for posterity, or in-case it helps anything (pls ignore if its useless)

This is where I need to write some documentation.
The short answer is no, the hardware can calculate it based on the other params.

I increased the dma_loop_buf to 16 * 16 which helped. It removed the artifacts at 33.MHz()

I'm glad this helped!

Just to be sure about the flickering, try changing the example to only draw on color and see if it still flickers.
You can do that by changing.

dpi.send(false, dma_buf).map_err(|e| e.0).unwrap();

to this

dpi.send(true, dma_buf).map_err(|e| e.0).unwrap();

[jgtaylor]

I can add to the review that I've tried this on the 4.3" cheap yellow board, at 800x480, and it looks wonderful. The 4.3" uses a ST7262.

The "flicker" i'd reported is definitely not flicker, but the backlight being a bit over-powered.

[Dominaezzz]

@MabezDev as requested, the example now runs on the official devkit.
I'd be very happy if this could make it into the next release.

[MabezDev]

Thanks for porting the example to the official devkit ❤️, unfortunately the devkit we have in the office is an older rev and the example isn't working.

I think we can still merge this as is for now, given that you have it tested, and we'll get a new rev of the board for testing later down the line.

[jessebraham]

Agree with the comment regarding #[non_exhaustive] on the Config struct, but otherwise changes LGTM I think

[Dominaezzz]

fwiw
https://github.com/user-attachments/assets/f142efc1-4d2a-4209-a85f-45edb63c9d61

[MabezDev]

Thank you!

[yanshay]

@Dominaezzz I can't find the example any longer in the latest examples folder, did it move?

I can add to the review that I've tried this on the 4.3" cheap yellow board, at 800x480, and it looks wonderful. The 4.3" uses a ST7262.

The "flicker" i'd reported is definitely not flicker, but the backlight being a bit over-powered.

@jgtaylor I also want to use ST7262, does the example work as is or did it require any changes?

[Dominaezzz]

It was moved. https://github.com/esp-rs/esp-hal/blob/main/qa-test/src/bin/lcd_dpi.rs

Also, note that I changed the example after jgtaylor's comment. So you'll need to update the FrameTimings

[Dominaezzz]

With Slint you may be able to implement bounce buffers (see esp-idf docs for explanation) and avoid PSRAM altogether.

That would mean pretty heavy load on the CPU, won't it? full time rendering. With complex UI it's not cheap.

I just remembered, the CPU load could be minimised by using DMA memcpy. It'll need some tuning to get right.

[yanshay]

I just remembered, the CPU load could be minimised by using DMA memcpy. It'll need some tuning to get right.

By 'bounce buffers' do you mean to render using slint at real time as the display is being fed with data using a small memory area of only several lines?
Or is there an option (which isn't slint specific) to have the entire screen in PSRAM and then do DMA from PSRAM to regular RAM and then DMA from regular RAM to display? Will DMA from PSRAM to regular RAM won't conflict with code execution from flash?

[Dominaezzz]

I just remembered, the CPU load could be minimised by using DMA memcpy. It'll need some tuning to get right.

By 'bounce buffers' do you mean to render using slint at real time as the display is being fed with data using a small memory area of only several lines? Or is there an option (which isn't slint specific) to have the entire screen in PSRAM and then do DMA from PSRAM to regular RAM and then DMA from regular RAM to display? Will DMA from PSRAM to regular RAM won't conflict with code execution from flash?

Bounce buffers work for both. The former would be ideal but I'm not sure if Slint will actually draw the screen fast enough to keep up with the LCD_CAM. Maybe with a bit of a head start it could work.

Code execution will impact the DMA from PSRAM to internal RAM, but if the bounce buffer is large enough, any slow downs should be absorbed by the buffer. I did get this to work with on the idf side, so it's certainly feasible. esp-hal is currently not exposing the DMA interrupts yet so you'll need the PAC to do this rn.

[yanshay]

Code execution will impact the DMA from PSRAM to internal RAM, but if the bounce buffer is large enough, any slow downs should be absorbed by the buffer. I did get this to work with on the idf side, so it's certainly feasible. esp-hal is currently not exposing the DMA interrupts yet so you'll need the PAC to do this rn.

Oh boy, that sounds complex...
As an alternative, who needs to be convinced to prioritize #2083?

[Dominaezzz]

It is complex indeed.

who needs to be convinced

@ MabezDev is the team lead, he's the one that needs convincing.
If you ask @ bjoernQ very very very nicely, he might do it in his free time 😄

[jgtaylor]

I've been working on trying to implement a bounce-buffer, but ... I've only manged to basically replicate the current DmaTxBuf, but with more steps and slower 😢 . I was looking into a DMA memcpy , however, I couldn't get all the parts to fit together (and I think I have some extra screws left-over!)

My latest effort has been to look into how it's done in the esp-idf with LVGL - they use a 256k bounce-buffer, but they also run code from PSRAM (which is #2083 ).

Sadly, unless I become a Rust expert very soon (improbable), I will merely continue to bang my head on my keyboard as I very, very, slowly get good 😀

[yanshay]

@jgtaylor How about adding your request for prioritizing #2083 on the issue discussion there?

[jessebraham]

Oh boy, that sounds complex... As an alternative, who needs to be convinced to prioritize #2083?

Given the tremendous amount of work we have ahead of us trying to get a stable release out, I would not get your hopes up. This is completely outside the scope of our current focus. Will probably be faster for you to implement it yourself and submit a PR.

[yanshay]

@jessebraham thanks for the update and setting expectations.

Will probably be faster for you to implement it yourself and submit a PR.

I wish I knew how to implement something like this. #2083 looks like way beyond my understanding of esp32. I'm more on the application programming side.
I'd be happy to assist in any way I can, like testing it under real world application etc.

Given the tremendous amount of work we have ahead of us trying to get a stable release out, I would not get your hopes up.

Can we ask for it to be considered for next release then? As I stated on the issue, it opens up large set of applications that are currently blocked from esp-hal. I think there's a strong justification for implementing it.

[Dominaezzz]

The esp-idf code snippet that bjeorn posted looks like it would do the trick.
I just don't have the time to type that in rust ATM.

[jessebraham]

Can we ask for it to be considered for next release then? As I stated on the issue, it opens up large set of applications that are currently blocked from esp-hal. I think there's a strong justification for implementing it.

When we have published the 1.0.0 release, updated the book and all other documentation, and ensured that our tooling is in reasonably good shape, then there will of course be discussions with regards to which drivers will be stabilized next.

Right now, we need to focus on creating a product that paying customers are able to use so that we can continue to justify an entire team of engineers being paid to work on this project. We have been discussing and planning this release for months already, so we need to focus on the scope of work which we have already committed to.

Realistically, our current plans will likely take us until late spring or early summer to complete. The community is welcome to contribute unstable drivers in the meantime and we will of course be happy to review them.

[yanshay]

Great to hear 1.0.0 is on the horizon! I wasn't aware of that.

Speaking of paying customers, I believe there's a significant commercial opportunity in GUI based embedded systems, and with Slint available (looks like they are building their revenue stream mostly around that) Rust seems to have the potential to become the easiest language for developing such applications.
Seems like the only thing standing between esp-hal and such applications is this relatively small feature because all available >3.5" displays require that.
But I'm not familiar with your business directions, so its only my 2 cents.

Anyway, I'll try sometime soon to take a stab at implementing this myself, maybe I'll be able to get something working.

Thanks!

[georgik]

@Dominaezzz @yanshay Hi guys! I figured our how to get full draw on 480x480 display with Rust no_std using dpi.rs and esp-hal 1.0.0. There are several gotchas.
It was pretty complex to capture proper configuration. The solution is using PSRAM to stream all data to screen.
I welcome suggestion or hints for improvement: https://github.com/georgik/esp32-conways-game-of-life-rs/tree/main?tab=readme-ov-file#esp32-s3-lcd-ev-board

[Dominaezzz]

Nice! Glad you were able to get DMA from PSRAM to keep up with the LCD_CAM. I'm a little surprised that the LCD pixels don't significantly discharge whilst the frame buffer is being prepared, perhaps I underestimate how fast the game update is.

For what you're doing, game of life, you'd probably be able to fit the entire frame buffer in internal RAM using a technique like #2238 .

[georgik]

@Dominaezzz @yanshay I've got no_std Slint working for 480x480 version:
https://github.com/georgik/slint/blob/feature/esp32-s3-lcd-ev-board/examples/mcu-board-support/esp32_s3_lcd_ev_board.rs

Code requires some polish, plus touch support. Yet, it's drawing :-)

[yanshay]

@Dominaezzz @yanshay I've got no_std Slint working for 480x480 version:

https://github.com/georgik/slint/blob/feature/esp32-s3-lcd-ev-board/examples/mcu-board-support/esp32_s3_lcd_ev_board.rs

Code requires some polish, plus touch support. Yet, it's drawing :-)

That's great!

What board is it exactly and what display does it use?

I have some 800x480 device, would this code work as is assuming it's the same display interface and just draw on part of the screen?

[georgik]

Touch is now working with FT5x06.

@yanshay 480x480 (GC9503CV) sub board is different from 800x480 (ST7262E43).
Details: https://docs.espressif.com/projects/esp-dev-kits/en/latest/esp32s3/esp32-s3-lcd-ev-board/user_guide.html#lcd-subboards

800x480 does not require that display initialization dance. RGB interface should be the same. According to ESP-BSP it should be simpler to get it running.

[yanshay]

I tried to get something working with the device I have (800x480). Just sending data to screen w/o slint for now.
I got it to work with the example available in esp-hal (tweaked the initialization to work), and it seems to be working.
Then I took some of the code that uses full frame DMA from PSRAM (continuous loop), and tried to just send a constant blue screen.
It sort of worked.
The screen looks blue, but with black dots, and not really clean blue.
So I recorded a few seconds of the blue frame from PSRAM (that looks bad) followed by the original example (which looks great).
Any ideas what could cause the PSRAM full frame not to work well?
Here is the video:

IMG_8827-2.mov

One thing that seemed strange to me, this is the FrameTiming I had to use to get the original example to work well.
It ended up being 830x480 rather than 800x480 to look good. Does it make sense?
In such case, should I use 800x480 as data or 830x480 as data? (both didn't look good with the blue psram frame)
All timing data I took from an arduino code for that display, so should be correct.

    config.timing = FrameTiming {
        horizontal_active_width: 830,
        horizontal_total_width: 860,
        horizontal_blank_front_porch: 8,

        vertical_active_height: 480,
        vertical_total_height: 510,
        vertical_blank_front_porch: 8,

        hsync_width: 4,
        vsync_width: 4,

        hsync_position: 10,
    };

[Dominaezzz]

Those black dots is caused by bandwidth issues, the DMA isn't able to keep up when using psram.

To fix/mitigate this, make sure psram is running at max speed, drop the DPI frequency to make it less demanding and increase the blank porches as much as you can. This will buy the DMA enough time to access psram.

No that 830 doesn't make sense, there's a good chance you haven't specified the porches correctly.
Don't copy it from the Arduino code directly, the names of the fields might look the same but they don't mean the same thing. You'll have to read the display datasheet and the FrameTiming docs to get it right.

[yanshay]

To fix/mitigate this, make sure psram is running at max speed

I know in the past it was possible to control the psram max speed (I actually added that), but later I remember esp-hal chose the high speed by default per chip type and I can't find an option to configure it. Is it available somewhere?

The PSRAM seems to me to be Octal (because with Quad it's even worse). Maybe I can modify the code to take advantage of that to get higher speed that isn't utilized with current code?

I tried everything you recommended and to me it seems that it's indeed the PSRAM throughput but I can't find how to improve on that and I'm sure it's possible because it is a common device used by other c++ applications w/o issues, and there's not enough memory to do that with only standard RAM.

, drop the DPI frequency to make it less demanding

I tried reducing frequency (using the original example that's working well). It worked inconsistently at lower speed.
Sometimes it worked at 15MHz. Then when I reduced again, it didn't work well.
The interesting thing is that when I increased frequency back even to the original 16Mhz it didn't work and I had to disconnect the device from power. As if it has some memory somewhere that prevents it from working.

and increase the blank porches as much as you can. This will buy the DMA enough time to access psram.

Aren't the blank porches at end of line? So if I see black dots on the same line it won't help?
Anyway, I tried and it didn't help

No that 830 doesn't make sense, there's a good chance you haven't specified the porches correctly.

The thing is that with 800, the right column pixel is flashing and not stable (in the original example that work, unrelated to PSRAM). Maybe it's because the display is shifted left for some reason due to configuration. If I use 801 its already showing good. I couldn't eliminate it and seems like data is missing for the last pixel. Maybe a dma configuration issue of missing one byte?

Don't copy it from the Arduino code directly, the names of the fields might look the same but they don't mean the same thing. You'll have to read the display datasheet and the FrameTiming docs to get it right.

This is what I have from the datasheet:

This how I mapped to code:

    let mut config = Config::default();
    config.clock_mode = ClockMode {
        polarity: Polarity::IdleLow,
        phase: Phase::ShiftLow,
    };
    config.frequency = 16.MHz();
    config.format = Format {
        enable_2byte_mode: true,
        ..Default::default()
    };
    config.timing = FrameTiming {
        horizontal_active_width: 800,
        horizontal_total_width: 840, // where to get from ? is that active_width + hsync_back_porch? or Period Time?
        horizontal_blank_front_porch: 10, // datasheet HSYNC Front Porch?  8 - 48

        vertical_active_height: 480,
        vertical_total_height: 504,  // where to get from ? is that active_height + vsync_back_porch?  or Period Time?
        vertical_blank_front_porch: 12, // datasheet: VSYNC Front Porch? 8 - 12

        hsync_width: 4, // Datasheet: VSYNC Pulse Width: 4 - 8
        vsync_width: 4, // Datasheet: HSYNC Pulse Width: 4 - 8

        hsync_position: 10, // where to get from?
    };
    config.vsync_idle_level = Level::High;
    config.hsync_idle_level = Level::High;
    config.de_idle_level = Level::Low;
    config.disable_black_region = false;

Most values don't have any meaningful effect, except horizontal_total_width which controls the side of the vertical band on the left that shows stable blue.
I think it's because this is the time where it 'blanks' before transferring data, so probably transfers the last data it had but not from PSRAM.

[Dominaezzz]

Is it available somewhere?

I'm not sure

The PSRAM seems to me to be Octal (because with Quad it's even worse). Maybe I can modify the code to take advantage of that to get higher speed that isn't utilized with current code?

If it's octal and running at 80MHz, then it's at max speed (probably).

but I can't find how to improve on that

Yeah, which is why I'm kinda surprised that it worked for @georgik

Aren't the blank porches at end of line? So if I see black dots on the same line it won't help?

At the end and at the start. During this time while the LCD_CAM isn't drawing pixels, the DMA can fill up its channel buffer from PSRAM.

Maybe it's because the display is shifted left for some reason due to configuration.

Yes, shifting happens when the DMA can't keep up and the LCD_CAM marches onward with black pixels. This makes then out of sync.

This is what I have from the datasheet

Max out all the blanking periods like this.

    config.timing = FrameTiming {
        horizontal_active_width: 800,
        horizontal_total_width: 800 + 48 + 48 + 8,
        horizontal_blank_front_porch: 48 + 8,

        vertical_active_height: 480,
        vertical_total_height: 480 + 12 + 12 + 8,
        vertical_blank_front_porch: 12 + 8,

        hsync_width: 8
        vsync_width: 8,

        hsync_position: 0, // This should almost always be 0
    };

If this doesn't work then I'm out of easy ideas.

[yanshay]

Thanks, that config didn't work.

I found the psramconfig struct has some settings:
In my case it's: PsramConfig { size: AutoDetect, core_clock: SpiTimingConfigCoreClock80m, flash_frequency: FlashFreq80m, ram_frequency: Freq40m }

I guess the only relevant one is the core_clock, increasing it made it worse, like data isn't read correctly.

Yeah, which is why I'm kinda surprised that it worked for @georgik

Do you mean to say that direct from PSRAM to RGB isn't supposed to work?
So how do all those lvgl based applications work with these devices?
They dma from psram to ram small pieces and then dma from ram to rgb?
Doesn't that create a different bottleneck from psram to ram?
Or maybe that's because they run code from Flash to RAM and then the PSRAM runs faster?

[Dominaezzz]

Do you mean to say that direct from PSRAM to RGB isn't supposed to work?

No, I'm saying esp-hal is missing features to make it work reliably.

[yanshay]

Do you mean to say that direct from PSRAM to RGB isn't supposed to work?

No, I'm saying esp-hal is missing features to make it work.

Is it #2083 ? Or something in addition to that?

[Dominaezzz]

There's more stuff.

Take a look at these

• https://docs.espressif.com/projects/esp-faq/en/latest/software-framework/peripherals/lcd.html
• https://esp32.com/viewtopic.php?t=26793

There are lots of little things that can be done to improve it.

[yanshay]

I just tried #2083 and it didn't help.
That's assuming I activated it correctly, I only psram_config.execute_from_psram = true;
I think it was activated because when not in Octal the code hang, and in Octal it worked.
I didn't improve on the display though, actually did not show even the slightest change in what is showing.
And it's not like I'm looking for performance now, just to display a solid color. So I think removing that obstacle should have made enough difference to make to get it to work, no?

It is so unfortunate that esp-hal doesn't support most of the displays out there... post release of 1.0 do you think it's going to be prioritized?

[georgik]

@yanshay Do you happen to have the code with blue screen public? I've finally got 800x480 board for testing.

[yanshay]

@yanshay Do you happen to have the code with blue screen public? I've finally got 800x480 board for testing.

The main.rs is available in #3024 (comment)

If you want, I could also push the repo to GitHub.

Please let me know how it goes. My board is JC8048W550, is it the one you have as well?

BTW @Dominaezzz - I've been thinking, would it be possible to use a fast SDCard instead of the PSRAM?
I know that they can get pretty fast, maybe faster than PSRAM. Optimized for sequential reads/writes and video.
Don't know if read/write to same area would cause wear on that type of memory.
I don't know how they perform for simultanious read/write though, and not how DMA from SPI to SPI is, and performance of the SPI interfaces, etc. WDYT? Would it conflict with the Flash reading as well?

[Dominaezzz]

(I never thought this PR's comment section would extend this long. At some point this topic should go into a discussion)

SD Card -> DPI, is theoretically possible, you'll just need to juggle the DMA descriptors accordingly.

SD Cards can't do read/write at the same time so I think you'll largely be stuck with read only workflows, unless you're fine with putting in some loading screens 😄

I can't speak on SD Card wear, but I'm not optimistic from the rumors I've heard. I suppose you could always acquire a high quality one.

SD Card in SPI mode won't be fast enough for the display. 25MHz (at 1 bit per cycle) won't keep up with the 16MHz (at 16 bits per cycle) needed for the display.
You'd need to use the SD/MMC peripheral to get fast enough (it can also do DMA).

There's also the optional of getting an external ram chip, separate from the built in one but I've never looked into this.

It would not conflict with flash as it's a different bus.