Merge pull request #770 from davidharrishmc/main

Started trickbox

Author: jordancarlin

docs/ctp/src/a.adoc

@@ -0,0 +1,51 @@
+
+=== A Atomic Extension
+
+The A extension is subdivided into Zaamo and Zalrsc.
+ <<t-A-normative-statements>> lists the normative statements in the RV{32/64}A specification applicable to all instructions, and the unprivileged coverpoints that exercise them.
+
+[[t-A-normative-statements]]
+.A Normative Statements
+[cols="1, 4, 2" options=header]
+|===
+|Unpriv Section|Normative Statement|Coverpoints
+|===
+
+<<t-A-priv-normative-statements>> summarizes normative statements relating to atomic instructions affecting privileged behavior.
+
+==== Zaamo Atomic Memory Operations Extension
+
+<<t-Zaamo-coverpoints>> summarizes the coverpoints for the Zaamo extension.
+
+[[t-Zaamo-coverpoints]]
+.Zaamo Instruction Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+include::{testplansdir}/Zaamo.csv[]
+,===
+
+[[t-Zaamo-normative-statements]]
+.Zaamo Normative Statements
+[cols="1, 4, 2" options=header]
+|===
+
+|===
+
+==== Zalrsc Load-Reserved / Store-Conditional Extension
+
+<<t-Zalrsc-coverpoints>> summarizes the coverpoints for the Zalrsc extension.
+
+[[t-Zalrsc-coverpoints]]
+.Zalrsc Instruction Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+include::{testplansdir}/Zalrsc.csv[]
+,===
+
+[[t-Zalrsc-normative-statements]]
+.Zalrsc Normative Statements
+[cols="1, 4, 2" options=header]
+|===
+|===

docs/ctp/src/aia.adoc

@@ -0,0 +1,54 @@
+
+=== AIA Advanced Interrupt Architecture
+
+This section describes AIA extension test plans.
+
+==== Smaia Machine-Mode Advanced Interrupt Architecture
+
+[[t-Smaia-coverpoints]]
+.Smaia Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/Smaia.csv[]
+,===
+
+==== Ssaia Supervisor-Mode Advanced Interrupt Architecture
+
+[[t-Ssaia-coverpoints]]
+.Ssaia Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/Ssaia.csv[]
+,===
+
+==== IMSIC Incoming MSI Controller
+
+[[t-imsic-coverpoints]]
+.IMSIC Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/imsic.csv[]
+,===
+
+==== APLIC Advanced Platform-Level Interrupt Controller
+
+[[t-aplic-coverpoints]]
+.APLIC Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/aplic.csv[]
+,===
+
+==== IOMMU I/O Memory Management Unit
+
+[[t-iommu-coverpoints]]
+.IOMMU Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/iommu.csv[]
+,===

docs/ctp/src/c.adoc

@@ -1,7 +1,7 @@
 
 === C Compressed Extension
 
-The C extension is subdivided into Zca, Zcf, and Zcd.  All configurations supporting C contain Zca.  RV32CF configurations also contain Zcf, and RV{32/64}CD configurations also contain Zcd.  A configuration supporting C should test all of the applicable Zc* extensions. <<t-C-normative-statements>> lists the normative statements in the RV{32/64}C specification applicable to all instructions, and the unprivileged coverpoints that exercise them.  Normative statements related to privileged behavior of C instructions are given in <<
+The C extension is subdivided into Zca, Zcf, and Zcd.  All configurations supporting C contain Zca.  RV32CF configurations also contain Zcf, and RV{32/64}CD configurations also contain Zcd.  A configuration supporting C should test all of the applicable Zc* extensions. <<t-C-normative-statements>> lists the normative statements in the RV{32/64}C specification applicable to all instructions, and the unprivileged coverpoints that exercise them.
 
 [[t-C-normative-statements]]
 .C Normative Statements
@@ -46,8 +46,6 @@ implementations that ignore them.|Hints are not tested
 include::{testplansdir}/Zca.csv[]
 ,===
 
-*** update all coverpoint refs
-
 [[t-Zca-normative-statements]]
 .Zca Normative Statements
 [cols="1, 4, 2" options=header]

docs/ctp/src/cbo.adoc

@@ -9,4 +9,5 @@ Certifying a RISC-V Device-Under-Test (DUT) involves the following steps:
 * Running the tests and reporting pass/fail
 
 Test developers also need to
-* Generate a coverage report confirming that the tests hit all the coverpoints footnote:[Open question whether to generate coverage from a Sail log or the DUT.  DUT requires a suitable testbench with RVVI interface, but is more robust to check that the tests fully run.]
+
+* Generate a coverage report confirming that the tests hit all the coverpoints based on a Sail trace

docs/ctp/src/cert.adoc

@@ -0,0 +1,135 @@
+
+== Architectural Functional Coverage
+
+In principle, a test plan could simply list a set of features to test, and a set of tests that exercise these features.  Experienced verification engineers have learned that tests often have bugs and don't exercise the feature that the author intended.  Many of these bugs can be caught if a different engineer independently writes functional coverpoints that measure whether the tests exercised the features.
+
+This Certification Test Plan focuses on whether a DUT implements all of the architectural features of a profile. Architectural features are those that are directly visible to the programmer, especially those defined in the unprivileged and privileged ISA specifications.
+
+[NOTE]
+====
+Later phases of certification tests may add ratified architectural features outside the ISA spec, such as IOMMU, debug, or AIA.
+====
+
+_Certification_ differs from _design verification_ in that it does not systematically test microarchitectural features. Design verification must be aware of DUT-specific features such as the pipeline design.  For example, DV should test design-specific interactions of interrupt arrival time, pipeline hazards, multicycle instructions, MMU, and memory system. Design verification also should test tricky implementation issues, such as hard floating-point rounding cases.
+
+This plan uses the term _Architectural Functional Coverage_ to describe the functional coverpoints required for certification.  In contrast, full design verification also has _Microarchitectural Functional Coverage_.
+
+_Functional coverage_ also differs from _code coverage_, which measures whether the test suite exercises all of the lines of code in the RTL.  Each of these detects different types of bugs.  Code coverage is part of DV but not certification.
+
+Architectural functional coverage requires a way to measure the architectural state of the machine after each instruction.  The https://github.com/riscv-verification/RVVI[RISC-V Verification Interface] (RVVI) provides a standard way to observe this state in the DUT.  This document defines Extended RVVI with additional signals to facilitate testing virtual memory.
+
+*** links to unpriv and priv specs
+
+
+=== Extended RVVI
+
+As compared to the base RVVI, Extended RVVI adds the following signals:
+
+[[t-extendedrvvi]]
+.Extended RVVI Signals
+[options=header]
+[cols="1, 1, 4" options=header]
+[%AUTOWIDTH]
+|===
+|Signal|Width|Description
+|virt_adr_i|XLEN|Virtual address of instruction
+|virt_adr_d|XLEN|Virtual address of data accessed by instruction
+|phys_adr_i|PA_BITS|Physical address of instruction
+|phys_adr_d|PA_BITS|Physical address of data accessed by instruction
+|pte_i|XLEN|Instruction page table entry
+|pte_d|XLEN|Data page table entry
+|ppn_i|PPN_BITS|Instruction physical page number
+|ppn_d|PPN_BITS|Data physical page number
+|page_type_i|2|Instruction page type ***explain
+|page_type_d|2|Data page type
+|read_access|1|Instruction reads data (loads, AMOs, ***CMO)
+|write_access|1|Instruction writes data (stores, AMO, ***cbo.zero)
+|execute_access|1|***define - is this jumps?  when asserted, and does it only affect data accesses?
+|===
+
+*** PA_BITS
+PPN_BITS = PA_BITS - 12
+*** why do we need ppn when we have phys_addr?  Why not derive locally?
+
+*** do we need to add guest physical address?  Such as
+
+* gpa_adr_i
+* gpa_adr_d
+
+maybe guest page table entries?
+
+=== riscvISACOV Signals
+
+Coverpoints are described using the open https://github.com/riscv-verification/riscvISACOV[riscvISACOV] framework.
+
+The framework collects data coming across the RVVI interface and presents it with `ins.current` and `ins.prev` data structures to allow easy access to the architectural state at the end of the current and previous instructions.  For example, `ins.current.rs1` is the source register for the current instruction, and `ins.current.rd_val` is the value written to the destination register by the current instruction.
+
+[NOTE]
+====
+Most coverpoints use `ins.current`, to measure architectural state at the end of the instruction.  However, some privileged coverpoints need to know the privilege mode or values of CSRs at the start of the instruction and thus use `ins.prev`.
+====
+
+The available fields are given in fcov/coverage/RISCV_coverage_rvvi.svh.  Most commonly used fields are summarized in <<t-riscvISACOV>>. Other fields include floating-point and vector register IDs and values, MMU extended RVVI signals, vector type information, and pending interrupts.
+
+[[t-riscvISACOV]]
+.riscvISACOV Signals for Coverpoints
+[options=header]
+[cols="1, 1, 4, 2" options=header]
+[%AUTOWIDTH]
+|===
+|Signal|Width|Description|Example
+|insn|ILEN|Instruction bit pattern|*** add x10, x11, x12
+|inst_name|string|Instruction name|add
+|disass|string|Disassembled instruction|`add x10, x11, x12`
+|trap|1|Instruction trapped rather than retiring|0
+|mode|2|Privilege mode 11 = machine, 01 = supervisor, 00 = user|11
+|pc_rdata|XLEN|PC of current instruction|0x80000000
+|pc_wdata|XLEN|PC of next instruction|0x80000004
+|rs1|5|Source register 1 ID|11
+|rs2|5|Source register 2 ID|12
+|rd|5|Destination register ID|10
+|rs1_val|XLEN|Source register 1 value|42
+|rs2_val|XLEN|Source register 2 value|69
+|rd_val|XLEN|Value written to destination register|111
+|imm|XLEN|Value of immediate|100
+|csr|XLEN x 4096|Value of CSRs|
+|===
+
+*** exact meaning of trap, importance in writing test
+*** do we need pending interrupts, or can this be part of mip?
+
+[NOTE]
+====
+RVVI and riscvISACOV support the notion of multiple harts that retire multiple instructions per cycle.  The number of instructions retired per cycle is a microarchitectural issue irrelevant to architectural functional verification, so it is set to 1, consistent with simulators that report each instruction retiring individually.  Phase 0-2 testing only involves a single hart.  Later phases may need to generalize to multiple harts to exercise fences and atomic operations that relate to memory consistency.
+====
+
+[NOTE]
+====
+Our riscvISACOV implementation uses a SystemVerilog disassmbler in fcov/disassemble.svh, replacing a proprietery disassembler from Imperas.  When adding new extensions, update the disassembler accordingly.
+====
+
+
+=== Writing Coverpoints with riscvISACOV
+
+*** example of coverpoint
+*** organization of files
+
+
+
+=== Sail Trace Coverage Flow
+
+Coverage is measured by running tests on Sail and generating a trace file, then passing this trace file to an RVVI agent in a SystemVerilog testbench that checks coverage.
+
+[NOTE]
+====
+The number of coverpoints and the number of tests both grow with the number of extensions, so the runtime to measure coverage can grow quadratically with the number of extensions, and becomes problematic.  To avoid this, it is recommended to only enable coverpoints for one extension (or small group of extensions) at a time and to only run tests for that extension, and generate a Unified Coverage Database (UDCB) file and report for that extension.  A script will kick off such tests in parallel for all extensions, and merge the coverage reports.
+====
+
+Sail produces the trace file in a standard RVVI-Trace format (see Section <<RVVI-Trace Format>>).  Any other simulator that implements the same trace format can also be used to cross-check coverage.
+
+*** Jordan add instructions on how to run this
+
+=== RVVI-Trace Format
+
+*** Jordan write
+*** how to handle instructions that change multiple CSRs, such as a write to mstatus.SIE also affecting sstatus.SIE

docs/ctp/src/coverage.adoc

@@ -64,6 +64,9 @@ include::profiles.adoc[]
 include::strategy.adoc[]
 include::unpriv.adoc[]
 include::priv.adoc[]
-include::nondeterministic.adoc[]
+include::nonisa.adoc[]
+include::trickbox.adoc[]
+include::coverage.adoc[]
+include::ref.adoc[]
 include::todo.adoc[]
 include::examples.adoc[]

docs/ctp/src/ctp.adoc

@@ -0,0 +1,34 @@
+
+=== Debug
+
+This section describes debug-related extensions not required in RVA23.
+
+==== DM Debug Module
+
+[[t-DM-coverpoints]]
+.DM Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/DM.csv[]
+,===
+
+==== DTM Debug Transport Module
+
+[[t-DTM-coverpoints]]
+.DTM Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/DTM.csv[]
+,===
+
+==== Sdext Debug Extension
+
+[[t-Sdext-coverpoints]]
+.Sdext Coverpoints
+[options=header]
+[%AUTOWIDTH]
+,===
+//include::{testplansdir}/Sdext.csv[]
+,===