N-Trace control interface and SBA

[algrobman]

Hi,

We are going to implement N-Trace encoder per hart in our CPU cluster. The N-Trace spec mandates to use SBA (System Bus Access) to access N-Trace registers/message RAM.
Our design had SBA option in the past, but it was deprecated because it was implemented to access only external to CPU memory, while our design had also local memories , accessible by Abstract Memory Commands (AMC) only. All this created difficulties for debugger to know how to access particular memory range. (Relatively easy solvable, having debugger source code, but it is real trouble for 3d party debugger vendors).

Seems it's not difficult to implement set of SBA registers to access N-Trace components ONLY, to be "compliant" with N-Trace spec , BUT will it work with "general" RISCV debuggers?

How will such debugger determine that the SBA works with N-Trace only and not suitable for system memories accesses?

See discussion beginning in:
riscv-non-isa/tg-nexus-trace#93

[pdonahue-ventana]

We are going to implement N-Trace encoder per hart in our CPU cluster. The N-Trace spec mandates to use SBA (System Bus Access) to access N-Trace registers/message RAM.

I'm surprised that it mandates SBA. If N-Trace isn't implemented or if a debugger doesn't support N-Trace, the debugger ends up needing to support SBA, abstract memory accesses, and the program buffer because nobody else mandates any one of the three.

Our design had SBA option in the past, but it was deprecated because it was implemented to access only external to CPU memory, while our design had also local memories

As you say, a particular address might access memory that's unique to the hart, meaning that the architectural state accessed by address X on hart 0 might be different than the architectural state accessed by address X on hart 1. That seems to be specifically considered in the SBA section of the spec when it talks about "access[ing] memory from a hart’s point of view" and how SBA might not be coherent (though non-coherent could also mean something else).

Seems it's not difficult to implement set of SBA registers to access N-Trace components ONLY, to be "compliant" with N-Trace spec , BUT will it work with "general" RISCV debuggers?

I didn't think that SBA is allowed to support only a subset of the address space, however I don't see anything that actually says that. It's at least a gray area so let's assume for this current discussion that SBA is allowed to support a subset.

Nothing warns debuggers that only a subset of the address space might be supported. My concern is that a debugger would see that SBA exists and then it would assume (like I did) that SBA can access all of memory. If it starts to get sberror=2 on almost everything, it would probably get very confused and not know that it should try to fallback to abstract memory accesses or the program buffer. It's not much different than your original problem where debuggers would get confused about the contents of address X.

Debuggers will assume a fairly straightforward implementation (every address is accessible and every address has one unique memory location) because there's no way for them to discover any implementation-dependent subtleties. There are 3 mechanisms to access memory and I'm pretty sure that no debugger is going to try to build up a map of what addresses work on what access mechanisms (only accepting that an address is inaccessible/nonexistent after all 3 mechanisms fail).

It's a gray area. If SBA is required to support everything then your idea isn't compliant. If SBA isn't required to support everything then I'm concerned that it's not actually useful. What are your thoughts?

[algrobman]

Paul,
I agree , with your analysis , but I don't understand how you, guys , create such inconsistent related specs (debug and N-trace) which are not going to work together . Are you reviewing work of each other?
Bad thing is that nobody sees full picture.

I'm a bit familiar with openocd RISCV support code. It checks existence of DM registers (SBA, ProgBuffer or Abstract command) selects one method and then uses it to access memory. You can instruct the tool what method to use from config file too.
But it can't select method dynamically . We used patched tool version when we had SBA and AMC to select access method by address range, but it did not work well because internal memories addresses and sizes were parameterizable by user.

Did you read the thread (link in my original comment)?
N-Trace maintainer insists that SBA is only compliant option to access N-Trace! and his biggest problem is that CPU triggers to control N-trace can't be programmed by external debugger while CPU is running!

this configuration with privet SBA was suggested by our designer of the N-trace module due the N-trace spec mandate (He also unfamiliar with RISCV debug subsystem, but I can't convince him that better just connect the module to the CPU as any other memory mapped internal device (PLIC or CLINT)). This way any memory access method would work.

Only cons is that the trace can't be controlled while the CPU is running, but I'm not convinced that is ultimate requirement

regarding:

how SBA might not be coherent (though non-coherent could also mean something else).

If the hart has data caches, it content can be modified vs external memory - SBA may access stale data. We deprecated SBA for that reason too.

[nadime15]

I'm a bit familiar with openocd RISCV support code. It checks existence of DM registers (SBA, ProgBuffer or Abstract command) selects one method and then uses it to access memory. You can instruct the tool what method to use from config file too.
But it can't select method dynamically

@algrobman I might be misunderstanding what you mean, but I am using telnet right now, and you can dynamically control how memory is accessed using the following command:

Must be one of: 'progbuf', 'sysbus' or 'abstract'.
riscv.cpu riscv set_mem_access method1 [method2] [method3]

You can trigger this at any point in time. You can write a script and pass it to GDB or Telnet to change the access method dynamically once certain conditions are met

[algrobman]

@nadime15, Are you saying that privet (only nTrace accessing) SBA is OK to implement?

[pdonahue-ventana]

I agree , with your analysis , but I don't understand how you, guys , create such inconsistent related specs (debug and N-trace) which are not going to work together . Are you reviewing work of each other?

What do you think is inconsistent between the two specs?

[nadime15]

@algrobman I was just pointing out that OpenOCD has a feature to dynamically change the memory access method at runtime. I wasn't commenting on whether a private/N-Trace-only SBA implementation is a good approach or not.

[algrobman]

how will upstream debugger know that it needs to use SBA for N-trace programming and Abstract command for everything else and to request to set openocd specific method?

[nadime15]

I am not familiar with the N-Trace specification, but based on what you have described, this is likely not possible at the moment without access to the source code of the debugger.

Must be one of: 'progbuf', 'sysbus' or 'abstract'.
riscv.cpu riscv set_mem_access method1 [method2] [method3]

I initially thought this command establishes a priority order for which methods to use, if the first method fails, it falls back to the second, and so on (this still needs to be verified). Assuming this is correct, it might make sense to check whether OpenOCD itself determines an order and tries each method sequentially (based on the available options). In that case, it could attempt SBA access for a given address, and if that fails, fall back to a second method (assuming it doesn’t stop entirely after the first error, which also needs to be confirmed).

[en-sc]

The N-Trace spec mandates to use SBA (System Bus Access) to access N-Trace registers/message RAM.

@algrobman, can you please provide a quote. I couldn't find this in the spec.

I have found this:
https://github.com/riscv-non-isa/tg-nexus-trace/blob/e0f02c6e4790097861111ac3679c6b43f0829463/docs/RISC-V-Trace-Control-Interface.adoc?plain=1#L370

External debuggers must be able to access all trace control registers independent of whether the traced harts are running or halted. That is why for external debuggers, the recommended access method for memory-mapped control registers is memory accesses through the RISC-V debug module using SBA (System Bus Access) as defined in the RISC-V Debug Specification.

From which it seems like SBA is recommended, not mandated.

I'm a bit familiar with openocd RISCV support code. It checks existence of DM registers (SBA, ProgBuffer or Abstract command) selects one method and then uses it to access memory. You can instruct the tool what method to use from config file too.
But it can't select method dynamically.

Could you please point me where are you getting this from?
AFAIK, RISC-V OpenOCD always tries all the methods from riscv set_mem_access in the order in which they were specified. By default: program buffer, then SBA, then AMC. If one method fails the next is attempted.

how will upstream debugger know that it needs to use SBA for N-trace programming and Abstract command for everything else and to request to set openocd specific method?

This can be implemented by modifying the order before the access and restoring it afterwards.
In general, a more high-level debugger can maintain a memory map for each process with the specifics on how to access each particular memory address (there is already something similar with flash memory).

[algrobman]

@en-sc , you had found the correct piece about N-Trace and SBA., The sentence with "External debugger must be able to access all trace control registers independent .." and the further statement that the debuggers must support N-trace programming thru SBA sound for me as mandating SBA for N-Trace programming if you want that your design to be compatible with 3d-party tools without modifications.

This can be implemented by modifying the order before the access and restoring it afterwards.
In general, a more high-level debugger can maintain a memory map for each process with the specifics on how to access each particular memory address (there is already something similar with flash memory).

All needed adjustments for particular HW implementation can be done relatively easy for any debugger if you have its source code. We patched openocd multiple times to make it work with our designs, but the goal of these standards ( debug and N-Trace) is to make sure that the debuggers and HW could be developed independently and still be compatible!

(on other hand, companies like Lauterbach are able to support almost all possible CPU architectures and debug interfaces with the same tool - Motorola several variants of PowerPC, M-core, ColdFires based chips, ARM, RISCV, MIPS, etc)
(Usually it takes them couple weeks to adopt new debug interface or architecture , based on my experience).

[en-sc]

This can be implemented by modifying the order before the access and restoring it afterwards.

@algrobman, I was a bit imprecise. This can be done without modifications to OpenOCD's source code through Tcl scripting.

[algrobman]

Any example how to use specific method based on address range? We also may have specific limitations on access size based on address range. ( the some local memories support only word size accesses, for ex, but debugger may want to place c.ebreak in there ..)

[en-sc]

This depends on which interface you use to access the memory.
There is not much you can do if you use GDB. However, if you are using Telnet / TclRPC to interact with OpenOCD you can call a user-defined Tcl procedure to access memory that wraps <targetname> read/write_memory commands and changes riscv set_mem_access according to some global address/size -> access method mapping.

The idea is something like:

proc read_memory_wrapped {address size} {
    set access_methods [riscv set_mem_access]
    riscv set_mem_access [get_access_method $address $size]
    set result [[target current] read_memory $address $size]
    riscv set_mem_access $access_methods
    return $result
}

I'm not sure it works (I don't remember what is the return of riscv set_mem_access and whether it can be used like this).
In any case, I hope the idea is clear. If some feature is missing in OpenOCD you can raise an issue in the OpenOCD repo.

[algrobman]

Thanks for the hint