How to create dma coherent memory on Nuclei Linux Platform

[matthewgui]

How to create dma coherent memory on Nuclei Linux Platform

The RISC-V Instruction Set Manual Privileged Architecture have added the Svpbmt Standard Extension for sv39,sv48,sv57, more details info refer to riscv standard doc. According to the doc, Svpbmt extension is only used for rv64.

Warning

Please make sure you follow the correct steps list below, if it is not working as expected, when you are looking for help from us, please let us know what is the difference you made compared to the doc below, otherwise it will be hard for you and us to investigate the root cause.

Nuclei 600/900/1000 series cpu cores can support riscv svpbmt extension and cmo extension. Please confirm whether the cpu version you have supports them.

core with svpbmt extension

Caution

• Kindly confirm with our AE and your hardware team regarding whether your CPU supports the Svpbmt extension and the CMO Extension.
• Svpbmt is not supported in rv32, only support rv64 now
• Please make sure dts changes applied to opensbi/uboot/kernel, just kernel is not enough, opensbi is the background runtime, it need to know hardware feature via dts, so it is really important.
• Please make sure you are using opensbi v1.3 version as we does
• It is recommended to make sure the align with the kernel/uboot/opensbi version as we used to avoid mis-aligned issue.

when cpu core support svpbmt extension, we can use this extension to create dma coherent memory area.

list patch based on nuclei sdk dev_nuclei_6.6_v2 branch.

Note

Tested on dev_nuclei_6.6_v2 branch, for other branches, please find similiar way by yourself.

• dts:

If SOC have no hardware dma coherent, dma-noncoherent dts properties need to be added.

dma-noncoherent;

add zicbom and svpbmt to isa, 64 is core cache line size(unit:byte), you can change it according to your hardware.

each core should attach with svpbmt to isa, and cbo block size.

riscv,isa = "rv64imafdc_zicbom_svpbmt"
riscv,cbom-block-size =<64>;

In theory,zicbom isa is not necessary,but from the perspective of kernel configuration dependencies and user scenarios, it is necessary to enable zicbom configuration.

dts_svpbmt.patch

• kernel config:

CONFIG_RISCV_ISA_ZICBOM=y
CONFIG_RISCV_ISA_SVPBMT=y

• toolchain:

At least this version of the toolchain can meet the requirements for selecting the CONFIG_RISCV_ISA_ZICBOM configuration.
nuclei gcc10 toolchain cannot meet gcc version requirements, while nuclei gcc13 toolchain can meet.

core without svpbmt extension

When cpu core have no svpbmt extension, we can first reserve a non-cachable memory area that meets the system requirements based on Nuclei mattri csr and use it in conjunction with the kernel global DMA pool to adapt kernel dma_alloc_coherent API.

list patch based on nuclei linux sdk dev_nuclei_6.6_v2 branch.

Note

Tested on dev_nuclei_6.6_v2 branch, for other branches, please find similiar way by yourself.

• opensbi:

we can use Nuclei mattri csr to reserve a non-cacheable memory area, please refer to Nuclei_RISC-V_ISA_Spec.pdf about mattri csr usage.

example:

reserve base:0xf0000000, size:0x100000 non-cachable area

static int nuclei_evalsoc_final_init(bool cold_boot, const struct fdt_match *match)) {
    ....

    #define mattri1_base 0x7f5
    #define mattri1_mask 0x7f6
    csr_write(mattri1_mask, 0xfff00000);
    csr_write(mattri1_base, 0xf0000005);

    return 0;
}

note:

mattri mask value: ~(size-1), area base address should be aligned area size.

Generally there are 5 sets of mattri CSR. If mattri CSR is used in multiple places, please allocate mattri CSR reasonably.

• dts:

If SOC have no hardware dma coherent, dma-noncoherent dts properties need to be added.

dma-noncoherent;

riscv,isa = "rv64imafdc_zicbom"
riscv,cbom-block-size =<64>;

if no cmo extension, Nuclei ccm extension is an alternative solution，Nuclei ccm commit refer to https://github.com/Nuclei-Software/nuclei-linux-sdk/commits/feature/5.10_ccm , Nuclei ccm doc refer to Nuclei_CCM_Mechanism.pdf

create reserved memory base:0xf0000000, size:0x100000 in dts node.

  reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;
    dam_coherent@f0000000 {
        compatible = "shared-dma-pool";
        reg = <0 0xf0000000 0 0x100000>;
        no-map;
        linux,dma-default;
    };
  };

above dts config is for rv64，rv32 can refer this to adapt.

rv32 reserve memory node

  reserved-memory {
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;
    dam_coherent@f0000000 {
        compatible = "shared-dma-pool";
        reg = <0xf0000000 0x100000>;
        no-map;
        linux,dma-default;
    };
  };

dts_no_svpbmt.patch

• kernel config:

CONFIG_DMA_DECLARE_COHERENT=y
CONFIG_DMA_GLOBAL_POOL=y

select CONFIG_DMA_GLOBAL_POOL to default y in kernel/dma/Kconfig

 config DMA_GLOBAL_POOL
    select DMA_DECLARE_COHERENT
    default y
    bool