Support binary log/trace output

[rsnikhil]

This first comment just describes the issue(s). Follow-up comments
discuss possible solutions.

Currently the Sail model writes out a log file in ASCII format. Issues:

• Logs can become very large (a binary format would be much smaller)

• Downstream tools (verification, trace-driven simulation, ...) have
  to parse ASCII text to recreate the log data

• We haven't been precise about defining a log format, and we update
  it as needed in an ad hoc fashion. As a result downstream tools
  break when we make such changes.

[rsnikhil]

I have looked at 4 public RISC-V specifications for "trace/log", and
none of them seem suitable for a full trace output from the Sail
model.

I recommend the Sail model use a 5th alternative, "Bluespec trace",
which I will discuss in a follow-up comment.

1. RVI E-Trace ("Efficient Trace")
   https://github.com/riscv-non-isa/riscv-trace-spec/releases/download/v2.0.3/riscv-trace-spec-asciidoc.pdf
   From RVI; ratified

2. RVI N-Trace ("Nexus Trace") https://github.com/riscv/tg-nexus-trace
   From RVI; ratified (nor nearing ratification)

3. RVFI ("RISC-V Formal Verification Interface")
   https://github.com/SymbioticEDA/riscv-formal From SymbioticEDA

4. RVVI ("RISC-V Verification Interface")
   https://github.com/riscv-verification/RVVI From Imperas originally
   (now part of Synopsys), but now public. Generated by the
   Imperas/Synopsys simulator. Used by OpenHWGroup.

E-Trace and N-Trace:

• focus on compressed control-flow trace (recording only
  "unpredictable" branch/jump/trap/xRET)

• rely on access to the original ELF file in order to reconstruct the
  full control trace

• are not suitable for self-modifying code/JIT where there is no ELF file

• do not capture arch-state updates (GPRs, FPRs, CSRs). E-Trace has a
  section for capturing memory updates.

RVFI and RVVI are similar (RVVI's README calls itself a kind of
superset of RVFI). They specify (nearly) all architectural state
updates that we'd need in a log, but they are not very suitable for
recording in a file, because it is VERY wide (e.g., all 32 GPR values,
all 32 FPR values, all defined CSR values, ...).

[rsnikhil]

Bluespec, Inc. has defined a trace model (PDF attached) which could be used by the Sail model.

Bluespec, Inc. is ready to contribute this spec to RVI as an open spec.

• It is a binary format
• Fields are optional, so can be dialled from minimal trace to verbose trace
• Can capture all standard arch state ("GC") updates (not yet spec'd for Vector)
• Can capture traps/interrupts
• Can capture additional intermediate state (e.g., CSR attempted and actual update,
  AMO pre- and post- values, ...)

This has been in use for some years inside Bluespec, Inc. for a number
of HW designs and simulators.

2020-03-10_trace-protocol.pdf

[rsnikhil]

Bluespec, Inc. is also happy to contribute to RVI a C program (which already exists) which reads and parses the binary trace and prints it in human-readable format.

This could also be restructured into a read-and-parse library + a human-readable-printer, so that the former can be used in other tools that need to read the trace.

[PeterSewell]

What format does the TestRig stuff use? Peter

…

On Mon, 9 Sept 2024 at 14:32, Rishiyur S. Nikhil ***@***.***> wrote: I have looked at 4 public RISC-V specifications for "trace/log", and none of them seem suitable for a full trace output from the Sail model. I recommend the Sail model use a 5th alternative, "Bluespec trace", which I will discuss in a follow-up comment. 1. RVI E-Trace ("Efficient Trace") https://github.com/riscv-non-isa/riscv-trace-spec/releases/download/v2.0.3/riscv-trace-spec-asciidoc.pdf From RVI; ratified 2. RVI N-Trace ("Nexus Trace") https://github.com/riscv/tg-nexus-trace From RVI; ratified (nor nearing ratification) 3. RVFI ("RISC-V Formal Verification Interface") https://github.com/SymbioticEDA/riscv-formal From SymbioticEDA 4. RVVI ("RISC-V Verification Interface") https://github.com/riscv-verification/RVVI From Imperas originally (now part of Synopsys), but now public. Generated by the Imperas/Synopsys simulator. Used by OpenHWGroup. E-Trace and N-Trace: - focus on compressed control-flow trace (recording only "unpredictable" branch/jump/trap/xRET) - rely on access to the original ELF file in order to reconstruct the full control trace - are not suitable for self-modifying code/JIT where there is no ELF file - do not capture arch-state updates (GPRs, FPRs, CSRs). E-Trace has a section for capturing memory updates. RVFI and RVVI are similar (RVVI's README calls itself a kind of superset of RVFI). They specify (nearly) all architectural state updates that we'd need in a log, but they are not very suitable for recording in a file, because it is *VERY* wide (e.g., all 32 GPR values, all 32 FPR values, all defined CSR values, ...). — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFMZZXLFH4UQ6FR6Y3HZGLZVWPOHAVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYGEZTQMRTGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[PeterRugg]

@PeterSewell In TestRIG we use RVFI, as documented here https://github.com/CTSRD-CHERI/TestRIG/blob/master/RVFI-DII.md#rvfi-dii-execution-packet-88-bytes. I guess a pro is that it's supported by the model already, though may need a little work to ensure there's nothing TestRIG specific.

but they are not very suitable for
recording in a file, because it is VERY wide (e.g., all 32 GPR values,
all 32 FPR values, all defined CSR values, ...)

That's certainly not true for RVFI as we implement it. Maybe that's new with RVVI? I can't see anything like that in https://github.com/SymbioticEDA/riscv-formal/blob/master/docs/rvfi.md either, except it does have optional ports per supported CSR, since lots of them can be updated as side-effects. Each packet is 88 bytes, though some updates, most notably CSR writes, are not directly reported in that trace.

Fields are optional, so can be dialled from minimal trace to verbose trace

That's a neat feature. We currently don't have that for TestRIG, and it would break the binary format to support.

Bluespec, Inc. is also happy to contribute to RVI a C program (which already exists) which reads and parses the binary trace and prints it in human-readable format.

We parse RVFI packets and print them in https://github.com/CTSRD-CHERI/QuickCheckVEngine/blob/master/src/QuickCheckVEngine/RVFI_DII/RVFI.hs, though that's in Haskell, and is mostly focussed on checking equivalence between two traces.

[rsnikhil]

Thanks for the link for TestRIG's RVFI spec (I had not seen that before).

That's certainly not true for RVFI as we implement it

You're right. In RVFI, a GPR update is defined as:

output [NRET *    5 - 1 : 0] rvfi_rd_addr
output [NRET * XLEN - 1 : 0] rvfi_rd_wdata

with 'rvfi_rd_addr' == 0 if Rd is not updated, (and NRET>1 only for superscalar retirement).

But it's still an XLEN+5-wide output even if Rd is not updated (signalled by 'rvfi_rd_addr' == 0)

For CSRs, there seems to be four XLEN-wide buses for each defined CSR.

---

In RVVI (file rvviTrace.sv), they have:

// X Registers
    wire [31:0][(XLEN-1):0]   x_wdata    [(NHART-1):0][(RETIRE-1):0];   // X data value
    wire [31:0]               x_wb       [(NHART-1):0][(RETIRE-1):0];   // X data writeback (change) flag

// Control and Status Registers
    wire [4095:0][(XLEN-1):0] csr        [(NHART-1):0][(RETIRE-1):0];   // Full CSR Address range
    wire [4095:0]             csr_wb     [(NHART-1):0][(RETIRE-1):0];   // CSR writeback (change) flag

i.e., all 32 registers, each with a valid bit (and similarly for FPRs), and all 4096 possible CSRs.

[rsnikhil]

We parse RVFI packets and print them in though that's in Haskell

(I like that! but) I think the Sail model audience's preference for such a reader is likely to be C and Python.

[allenjbaum]

What we want from SAIL is, for each instruction executed, - the instruction fetch address and data (along with an implicit memory accesses ( PTEs) with their level and whether it is a read or write (A/D bit accesses) There is currently a corner case in Sail now when there is an instruction access fault, where CSR updates aren't exposed the disassembled instruction is rather invaluable for debugging - but might make coverage easier to write - Each register/CSR number(s) of any explicit (e.g. Rd) or implicit (generally CSR side effects) writes, along with the value update, Note that this has to deal with multiple registers begin updated by an op (unaligned ld/st, register pairs, vector load/store), and all register types (Xreg/ Vregs, CSRs) - all data memory reads or write addresses Note that this has to deal with multiple access (as above) We certainly don't want to see all state at each instruction boundary, just the changes. The Sail log also outputs mode changes (interrupts/exceptions/returns). We can probably figure that out by looking at the CSR updates (e.g. xCause being written), but there are corner cases that make it difficult. NRET is probably not applicable for SAIL.

…

On Mon, Sep 9, 2024 at 8:25 AM Rishiyur S. Nikhil ***@***.***> wrote: We parse RVFI packets and print them in though that's in Haskell (I like that! but) I think the Sail model audience's preference for such a reader is likely to be C and Python. — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJVJWFI67CFV2JMRSGDZVW4YJAVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYGQZDINJSGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jrtc27]

For RVFI it's worth noting that what's in the spec is heavily aimed towards the physical hardware interface, where it's simpler to have a whole bunch of wires that may or may not be used. Having a smarter encoding of that same data for software would make a lot of sense (e.g. not encoding wdata if rd is 0).

[PeterSewell]

@Alasdair Armstrong ***@***.***> should we just use (a handy representation of) new-interface events?

…

On Mon, 9 Sept 2024, 19:53 Jessica Clarke, ***@***.***> wrote: For RVFI it's worth noting that what's in the spec is heavily aimed towards the physical hardware interface, where it's simpler to have a whole bunch of wires that may or may not be used. Having a smarter encoding of that same data for software would make a lot of sense (e.g. not encoding wdata if rd is 0). — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFMZZQB27O6B5RYBMWDY4TZVXU23AVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYHA2DQNRXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[allenjbaum]

I completely agree. But, you still need a tag to identify the regieter number and which kind of register it is (currently Xreg, Vreg, and CSR; soon maybe Matrix regs) That's just now, and we'll probably need to look ahead to 48/64b instructions, where the number of Vregs might expand, there may be separate Mask regs, etc_. So, right now it's a minimal 5bitreg# + 2bit tag, though you could compress out the tag to a single bit if you're clever. Not sure it is worth the bother. It might be easiest to just define an 8b tag to simply implementation.

…

On Mon, Sep 9, 2024 at 11:51 AM Jessica Clarke ***@***.***> wrote: For RVFI it's worth noting that what's in the spec is heavily aimed towards the physical hardware interface, where it's simpler to have a whole bunch of wires that may or may not be used. Having a smarter encoding of that same data for software would make a lot of sense (e.g. not encoding wdata if rd is 0). — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJTP4WDSR7X73D6O7DLZVXU23AVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYHA2DQNRXGI> . You are receiving this because you commented.Message ID: ***@***.***>

[Timmmm]

I had a brief skim through Bluespec's format. It looks pretty well designed. I don't think we can use it as-is though, and I would favour an encoding that was more standard - I think Protobuf is probably the best choice.

I think there are three things that have to happen:

1. We need to complete Implementing callbacks in sail-riscv for state-changing events #449 (state callbacks).
2. We need to define the semantics of what is captured in the binary log.
3. We need to define the encoding.

With regard to the semantics, I can comment on some random things I've learned from my experience with Codasip's system:

1. Registers aren't just 32/64-bit. Vector registers can be very long. There's also CHERI (65 or 129 bits). Basically you need to be able to store an arbitrarily long bit vector.
2. You need to be able to store any number of register writes and memory accesses in each step.
3. Store register identities by index and type (i.e. X, 5, CSR, 53, V, 14) not by name (mstatus, x3).
4. Our system doesn't do this, but it would be extremely helpful if register writes also have a flag to say whether or not the value written to registers actually changed them. Otherwise you find yourself having to align models on things like whether fflags gets recorded as a register write in different situations.
5. It's very helpful to record physical and virtual addresses for each memory operation, and also PMAs but that's a little trickier since they aren't standard.
6. We calculate architectural coverage from the log file, however one thing that makes it difficult is that it doesn't store register input values for instructions. Technically that could be redundant information - you can maybe go back in the trace and figure it out - but that's a bit painful. It would be much nicer if you could just record that information directly in each step.

[allenjbaum]

I believe ISAC keeps a running "cache" of register values (of all types based on the immediately previous write) and uses that if necessary. I'd object if we had to expand the size of the log with that information, because it would expand it quite a lot. I hadn't thought about exposing the VA; that can be extracted from the running log by using the previous value of the basereg and offset, at the expense of parsing the opcode. (which SAIL already does for you in the disassembly, right?). I Im not sure where the 53 comes from in the above - the are 4K potential indices, plus the indirect ones (which, can be kept track of by the previous write of the select CSR). I'm not sure where the 14 comes from either: there are 32 vector registers.

…

On Sun, Sep 15, 2024 at 2:08 PM Tim Hutt ***@***.***> wrote: I had a brief skim through Bluespec's format. It looks pretty well designed. I don't think we can use it as-is though, and I would favour an encoding that was more standard - I think Protobuf is probably the best choice. I think there are three things that have to happen: 1. We need to complete #449 <#449> (state callbacks). 2. We need to define the semantics of what is captured in the binary log. 3. We need to define the encoding. With regard to the semantics, I can comment on some random things I've learned from my experience with Codasip's system: 1. Registers aren't just 32/64-bit. Vector registers can be very long. There's also CHERI (65 or 129 bits). Basically you need to be able to store an arbitrarily long bit vector. 2. You need to be able to store any number of register writes and memory accesses in each step. 3. Store register identities by index and type (i.e. X, 5, CSR, 53, V, 14) not by name (mstatus, x3). 4. Our system doesn't do this, but it would be extremely helpful if register writes also have a flag to say whether or not the value written to registers actually changed them. Otherwise you find yourself having to align models on things like whether fflags gets recorded as a register write in different situations. 5. It's very helpful to record physical and virtual addresses for each memory operation, and also PMAs but that's a little trickier since they aren't standard. 6. We calculate architectural coverage from the log file, however one thing that makes it difficult is that it doesn't store register input values for instructions. Technically that could be redundant information - you can maybe go back in the trace and figure it out - but that's a bit painful. It would be much nicer if you could just record that information directly in each step. — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJWQDTTNSAMTUJTVV23ZWXZMBAVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRG44TEMBXHA> . You are receiving this because you commented.Message ID: ***@***.***>

[Timmmm]

Yes all this stuff can often be figured out from the previous events, but it makes it a fair bit more complex and fragile. But I agree with your approach that most things should be optional so we can just make this information optional too if people want to make that trade-off.

By the way I forgot one extra thing:

7. You need a "reset" event, and that event should include the writes due to reset. The trace should begin with one of those events, and the first one should include the values of all registers.

Im not sure where the 53 comes from in the above

Those were just examples of register indices.

[allenjbaum]

RISC-V explicit limits reset values to be undefined unless otherwise stated - and there are very few registers that have defined reset values. Implementations, on the other hand, almost always define the reset value - and (in the ACT case at least), the config file can lists those reset values. The intent is that SAIL read the config file and initialize those reset values (or leave them undefined), so those don't need to be output in the log. - they're constants. Boot code may want/need to explicitly initialize otherwise uninitialized CSRs, and in that case the results of that code would naturally be output in the trace.

…

On Mon, Sep 16, 2024 at 1:05 AM Tim Hutt ***@***.***> wrote: Yes all this stuff can often be figured out from the previous events, but it makes it a fair bit more complex and fragile. But I agree with your approach that most things should be optional so we can just make this information optional too if people want to make that trade-off. By the way I forgot one extra thing: 7. You need a "reset" event, and that event should include the writes due to reset. The trace should begin with one of those events, and the first one should include the values of *all* registers. Im not sure where the 53 comes from in the above Those were just examples of register indices. — Reply to this email directly, view it on GitHub <#545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPXVJX5FPASRNL7C6NYLILZW2GNLAVCNFSM6AAAAABN4RY7TWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJSGI2TINBSGU> . You are receiving this because you commented.Message ID: ***@***.***>