[Nexys4 DDR] Struggling to load into ICCM (address 0xEE000000)

[mablinov]

Hi all,

I've been interested in running some tests on the SweRVolf, and have been succesfully programming it within the RAM region (0x00000000-0x07FFFFFF) and the DCCM region (0xF0040000-0xF004FFFF).

However, as I understand the EH1 core also has a memory region called "ICCM" for fast instruction access, at (0xEE000000-0xEE00FFFF).

The DCCM works quite well: Infact I've already got some figures for the memory bandwidth:

code in RAM, data in RAM: read @ 21.5MB/s, write @ 25MB/s
code in RAM, data in DCCM: read @ 136MB/s, write @ 136MB/s

(p.s. not sure why RAM is slower to read than to write, but anyway...)

What I'd like to do now is to put my code into ICCM - but I'm having issues loading it. Here is my gdb input:

(gdb) tar rem :3333
Remote debugging using :3333
_finish () at crt0.s:32
32              beq x0, x0, _finish
(gdb) load
Loading section .text.init, size 0x30 lma 0x10000
Loading section .text, size 0x812 lma 0x10030
Loading section .data, size 0xaf lma 0x20000
Loading section .text.iccm, size 0x1a lma 0xee000000
Start address 0x00010000, load size 2315
Transfer rate: 10 KB/sec, 578 bytes/write.
(gdb)

And here are the OpenOCD errors that I see after issuing the load gdb command:

Info : accepting 'gdb' connection on tcp/3333
Error: Target riscv.cpu: Failed to write memory (addr=0xee000000)
Error:   progbuf=disabled, sysbus=disabled, abstract=failed

If I try memory dumping the location, I get

(gdb) x/8x 0xee000000
0xee000000 <run_in_iccm>:       0x00000000      0x00000000      0x00000000      0x00000000
0xee000010 <run_in_iccm+16>:    0x00000000      0x00000000      0x00000000      0x00000000

and OpenOCD also spews out more errors at me:

Error: Target riscv.cpu: Failed to read memory (addr=0xee000000)
Error:   progbuf=disabled, sysbus=disabled, abstract=failed
Error: Target riscv.cpu: Failed to read memory (addr=0xee000004)
Error:   progbuf=disabled, sysbus=disabled, abstract=failed
...(trimmed)...
Error: Target riscv.cpu: Failed to read memory (addr=0xee00001c)
Error:   progbuf=disabled, sysbus=disabled, abstract=failed

The OpenOCD launch command is openocd -f swervolf_nexys_debug.cfg and the swervolf_nexys_debug.cfg is the default one supplied with the repo (md5sum: af7ea2b7922e9b34f8e578c18bbccfbd)

openocd --version report: (p.s. Ignore the bb96dc521 - that is my commit to fix the texinfo documentation build on my old ubuntu. the actual last meaningful commit is 5215fc52a Fix flashing on HiFive1. (#649))

Open On-Chip Debugger 0.11.0+dev-02000-gbb96dc521 (2021-10-08-15:57)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html

Any ideas? I'm pretty sure there is an ICCM present, because if I look into the following file in the core build directory (build/swervolf_0.7.4/nexys_a7-vivado/config/common_defines.vh), I find

...(stripped output)...
`define RV_ICCM_SADR 32'hee000000
`define RV_ICCM_OFFSET 10'he000000
`define RV_ICCM_BANK_BITS 3
`define RV_ICCM_INDEX_BITS 14
`define RV_ICCM_SIZE_512 
`define RV_ICCM_EADR 32'hee07ffff
`define RV_ICCM_ROWS 16384
`define RV_ICCM_NUM_BANKS 8
`define RV_ICCM_RESERVED 'h1000
`define RV_ICCM_SIZE 512
`define RV_ICCM_NUM_BANKS_8 
`define RV_ICCM_BITS 19
`define RV_ICCM_REGION 4'he
`define RV_ICCM_DATA_CELL ram_16384x39

So it's definitely at 0xee000000, and it should be present, but I'm struggling to load into it. Should I be manually enabling the ICCM?

[mablinov]

Some notes:

Running OpenOCD under debug confirms that upon sending the command to execute a write to the address of 0xee000000, the abstractcs value returned is 0x502. Looking at the source code for SweRV, bits [10:8] of the abstractcs register are reserved for errors. From 0x502, that corresponds to bits [10:8] being 0x5, which looking at the source in dbg/dbg.sv corresponds to a bus error:

   assign        abstractcs_error_din[2:0]  = abstractcs_error_sel0 ? 3'b001 :                  // writing command or abstractcs while a command was executing. Or accessing data0
                                                 abstractcs_error_sel1 ? 3'b010 :               // writing a illegal command type to cmd field of command
                                                    abstractcs_error_sel2 ? 3'b011 :            // exception while running command
                                                       abstractcs_error_sel3 ? 3'b100 :         // writing a comnand when not in the halted state
                                                          abstractcs_error_sel4 ? 3'b101 :      // Bus error
                                                             abstractcs_error_sel5 ? 3'b111 :   // unaligned or illegal size abstract memory command
                                                                abstractcs_error_sel6 ? (~dmi_reg_wdata[10:8] & abstractcs_reg[10:8]) :   //W1C
                                                                                        abstractcs_reg[10:8];                             //hold

abstractcs_error_sel4 is itself set as a function of some internal bus states, of which I can't make much:

...(trimmed)...
            SB_CMD_RESP: begin
                     dbg_nxtstate         = dmcontrol_reg[1] ? IDLE : CMD_DONE;
                     dbg_state_en         = ((sb_axi_rvalid & sb_axi_rready) | (sb_axi_bvalid & sb_axi_bready)) & dbg_bus_clk_en;
                     dbg_sb_bus_error     = ((sb_axi_rvalid & sb_axi_rready & sb_axi_rresp[1]) | (sb_axi_bvalid & sb_axi_bready & sb_axi_bresp[1])) & dbg_bus_clk_en;
                     data0_reg_wren2      = dbg_state_en & ~sb_abmem_cmd_write & ~dbg_sb_bus_error;
            end
...(trimmed)...

[mablinov]

HI @olofk, I was wondering if perhaps you're more familiar with the core and have any ideas?

[aprnath]

Hi @mablinov, are you explicitly building yor SweRVolf with ICCM enabled? The default build (see configs/swerv.config) is to build with ICCM disabled (i.e not present).
The breadcrumb to look for in the generated defines file is RV_ICCM_ENABLE, with a value of 1. If this define is not present, it indicates that there is no ICCM in the core being configured. In your grep of ICCM in the common_defines.vh file, there is no RV_ICCM_ENABLE present.

Without an ICCM, the address provide will be steered towards the system bus - where here is probably no memory attached at this address - hence the bus error indication.