Running DOOM on NEORV32

[vogma]

Hey!

I've been having a lot of fun with the NEORV32 on my Arty A7-100 FPGA lately. I recently got the DDR3 SDRAM hooked up to the NEORV32 over AXI, using it for both code and data. This meant I suddenly had 256 MB at my disposal. Naturally, that got me wondering what I could do with all that space. And since DOOM gets ported to pretty much everything, I figured why not to this RISC-V soft-core?

I wrote a VGA subsystem in VHDL for this. There's a DMA engine that pulls the framebuffer out of DDR3, and a VGA controller that pushes out 640x480 @ 60 Hz through a PMOD connector. The WAD file gets loaded off an SD card over SPI at boot, and the game renders into a double-buffered framebuffer sitting in DDR3. Right now it's running at around 2.4 FPS.

https://github.com/vogma/neodoom

doom.mp4

[dthomas981]

This is very cool, well done!

[stnolting]

I've always wanted to port Doom... It's practically a must-have. 😅 So this is really really cool! 👍

2.4 FPS sounds pretty good. What clock speed is the CPU running at?

I see that you have quite a few ISA extensions enabled. Does this result in a noticeable performance gain? And do you really need the FPU?

Where is the current bottleneck in terms of FPS?

[vogma]

Thank you @stnolting :)

The cpu is running at 83.33 MHz. It is the clock derived from the Memory Interface Controller (MIG) which interfaces the SDRAM. Everything is clocked from the MIG except the VGA controller.

I have now disabled the FPU in the block design, doom is integer/fixed-point only and also the Makefile didn't use the extension. I capture HPM counters for each frame and print telemetry over UART every 60 frames.

The bottleneck is write-through stores to DDR3. Each of the 256k per-frame framebuffer writes stalls the CPU ~39 cycles because there's no store coalescing. A hardware store buffer would probably result in the biggest performance gain.

Doom is running a continuous loop of game logic, rendering, and frame buffer conversion. During rendering, it writes the frame into a internal framebuffer I_VideoBuffer (320x200, 8-bit indexed color). In fbconv the internal buffer is read, converted to RGB565, scaled by a factor of two and written back to sdram. These numbers show the avg. amount of cycles per frame the NEORV32 spents in each block:

phase   cycles  loads   stores  lsu%   alu%  disp%  frame%
logic     1M      50k      9k   38.5   2.9   11.0   3.8%
render    7M     283k     93k   50.3   2.3   5.5    20.8%
fbconv   25M    1169k    256k   39.1   3.5   7.9    75.3%
---------------------------------------------------------
TOTAL    33M    1503k    359k   41.4   3.2   7.5

The NEORV32 spends 75% of cycles for each frame in fbconv. During that time the LSU is in waiting for memory 39% of the time.

To isolate store stalls from load stalls during fbconv, i re-measured fbconv with framebuffer stores
replaced by an inline asm barrier (asm volatile("" :: "r"(val))) that forces the
compiler to preserve the full pixel computation pipeline (palette lookup, RGB565 packing)
while emitting zero actual store instructions. This eliminates only DDR3 write traffic during that phase:

fbconv          cycles  loads   stores  lsu%   alu%  disp%
normal           25M    1169k    256k   39.1%  3.5%   7.9%
skip stores      15M    1164k      0k   13.0%  4.2%  13.1%
delta (stores)   10M

The loads are nearly identical so the computation pipeline should be fully preserved.

	Cycles	% of normal fbconv
LSU stalls from loads	~1.95M (15M × 13.0%)	7.8%
LSU stalls from stores	~7.8M (9.8M − 1.95M)	31.3%

So even though the fbconv phases issues 4.5x the amount of reads compared to writes, the LSU stalls for the writes 4x longer than for the reads.

[stnolting]

Thanks fpr the info!

The cpu is running at 83.33 MHz.

Can you say something about the critical path? On an FPGA like this, an even higher clock speed should be possible, at least for the core logic. 🤔

The bottleneck is write-through stores to DDR3. Each of the 256k per-frame framebuffer writes stalls the CPU ~39 cycles because there's no store coalescing. A hardware store buffer would probably result in the biggest performance gain.

Is this handled by the CPU (i.e., by the software), or do you use something like a DMA for this?

I once experimented with simple write buffers for the data cache. The performance gain was quite modest, and the "out of order" behavior - especially in the case of bus errors - resulted in considerable hardware overhead. That's why I'm currently working (once again) on a full write-back architecture for the cache. Let's see how that turns out.

In fbconv the internal buffer is read, converted to RGB565, scaled by a factor of two and written back to sdram.

That sounds like a lot of shift and masking operations or LUT lookups. How about a custom instruction for that?

The NEORV32 spends 75% of cycles for each frame in fbconv. During that time the LSU is in waiting for memory 39% of the time.

Does it wait for cache blocks to be loaded (read misses) or does it wait for write-through accesses to be completed?

So even though the fbconv phases issues 4.5x the amount of reads compared to writes, the LSU stalls for the writes 4x longer than for the reads.

Ah, okay, that's a lot, of course. The frame buffer is processed in stages, right? That could really help with caching for write accesses... Have you tried connecting an AXI cache in front of the MIG?