RISC-V Opcodes Data Cross-Validation

[AFOliveira]

I integrated @BrianAnakPintar with normal UDB directory for this. I also added the generation of riscv-opcodes outputs, such as encodings.out.h that is what @ayosher was looking for if I am not in error. I added a README.md for detailed instructions.

@ayosher Until next meeting I won't have enough bandwidth to test this with SPIKE and Qualcomm new extensions, but to do so, you should just go to the folder /ext/opcodes_maker and run make YAML_DIR=../../cfgs/qc_iu/arch_overlay/inst/ and look for output/encoding.out.h as an input for SPIKE.

[ThinkOpenly]

Before you merge, I ask that you change:

Integrate Brain branch ...
to
"Integrate Brian branch ..."

Although, @BrianAnakPintar is a really smart guy. ;-)

[AFOliveira]

@ThinkOpenly Ups! I'll do so :)

[dhower-qc]

• Can we move this to backends/ rather than ext/?
• Did you mean to include instr_dict.json? If so, we should figure out how to make sure the checked-in version is always consistent with the state of the database.

[AFOliveira]

1. Sure!
2. If I did it was by accident. Default Path takes us to arch/inst and configurable user path can use whatever folder, like cfgs/qc_ui/inst

[dhower-qc]

Cool. Now let's just make sure this gets added to the smoke tests. Currently, that takes two changes:

• Update test:smoke task in Rakefile (for local runs)
• Update .github/workflows/regress.yml (for GitHub runs)

Now that you've told me there is a tool to run GitHub workflows locally, we should be able to get rid of step 1 eventually...

[AFOliveira]

@dhower-qc This is only generating riscv-opcodes outputs, therefore, shouldn't this be a separate task and not be inside the smoke test? This does not test anything, only generates. Probably I could put it into some gen:opcodes_outputs task?

[ayosher]

I integrated @BrianAnakPintar with normal UDB directory for this. I also added the generation of riscv-opcodes outputs, such as encodings.out.h that is what @ayosher was looking for if I am not in error. I added a README.md for detailed instructions.

@ayosher Until next meeting I won't have enough bandwidth to test this with SPIKE and Qualcomm new extensions, but to do so, you should just go to the folder /ext/opcodes_maker and run make YAML_DIR=../../cfgs/qc_iu/arch_overlay/inst/ and look for output/encoding.out.h as an input for SPIKE.

I will try to run it. I hope it is merged with latest main branch to include all fixes to Xqci/Xqccmp. Now dealing with building environment to work on Spike, will take few days. Would be great to see this pull request in main...

[AFOliveira]

@ayosher Just merged this with main to ensure you have the most recent version.
There is an FYI to be made that UDB and riscv-opcodes don't describe all instructions in the same manner, so some parts of the output are different from what Spike usually intakes, this is more related to the V extension so it likely won't affect your use case. There is need for a bit of discussion on this, so I'll bring it up in the next meeting.

[dhower-qc]

@dhower-qc This is only generating riscv-opcodes outputs, therefore, shouldn't this be a separate task and not be inside the smoke test? This does not test anything, only generates. Probably I could put it into some gen:opcodes_outputs task?

Sorry, I meant test:regress 😄

[ayosher]

Should I expect down the road this generation to be integrated in some "do" task?

[AFOliveira]

Should I expect down the road this generation to be integrated in some "do" task?

That is what I was discussing with Derek. I would say yes, this is an output as many others are; however, I'm not entitled to answer it with certainty by myself. @dhower-qc any thoughts on it?

[ayosher]

I tried to generate opcodes for qc_iu/Xqci, faced the following error message:
Successfully processed 157 YAML files Output written to: output/instr_dict.json python3 generator.py output/instr_dict.json -c -chisel -spinalhdl -sverilog -rust -go -latex Traceback (most recent call last): File "/quic/riscv-unified-db/backends/opcodes_maker/generator.py", line 33, in <module> from c_utils import make_c ModuleNotFoundError: No module named 'c_utils' make: *** [Makefile:38: generate] Error 1

[dhower-qc]

absolutely. we should have something like do gen:opcode_header.

[AFOliveira]

@ayosher riscv-opcodes was not updated in remote branch. My bad. There should not be any errors now.

[AFOliveira]

Updated Rakefile. Example usage:

./do gen:opcode_outputs YAML_DIR=cfgs/qc_iu/arch_overlay/inst/

YAML_DIR is optional and can be left blank to default to normal arch/inst. I'll add to regression test later.

[ayosher]

Updated Rakefile. Example usage:

./do gen:opcode_outputs YAML_DIR=cfgs/qc_iu/arch_overlay/inst/

YAML_DIR is optional and can be left blank to default to normal arch/inst. I'll add to regression test later.

Tried that. I still have the same issue. Now with ./do way.
I believe I lack some package somewhere - but I have no idea which one...

Successfully processed 157 YAML files
Output written to: /quic/riscv-unified-db/gen/opcodes_outputs/instr_dict.json
/quic/riscv-unified-db/.home/.venv/bin/python3 /quic/riscv-unified-db/backends/opcodes_maker/generator.py /quic/riscv-unified-db/gen/opcodes_outputs/instr_dict.json -c -chisel -spinalhdl -sverilog -rust -go -latex
Traceback (most recent call last):
File "/quic/riscv-unified-db/backends/opcodes_maker/generator.py", line 33, in
from c_utils import make_c
ModuleNotFoundError: No module named 'c_utils'
/quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/file_utils.rb:65:in block in create_shell_runner': Command failed with status (1): [/quic/riscv-unified-db/.home/.venv/bin/python3 /quic/riscv-unified-db/backends/opcodes_maker/generator.py /quic/riscv-unified-db/gen/opcodes_outputs/instr_dict.json -c -chisel -spinalhdl -sverilog -rust -go -latex] (RuntimeError) from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/file_utils.rb:57:in sh'
from /quic/riscv-unified-db/Rakefile:49:in block (2 levels) in <top (required)>' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in block in execute'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in each' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:281:in execute'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:219:in block in invoke_with_call_chain' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in synchronize'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:199:in invoke_with_call_chain' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/task.rb:188:in invoke'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:188:in invoke_task' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in block (2 levels) in top_level'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in each' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:138:in block in top_level'
from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:147:in run_with_threads' from /quic/riscv-unified-db/.home/.gems/ruby/3.2.0/gems/rake-13.2.1/lib/rake/application.rb:132:in top_level'
from -e:1:in `

'

[AFOliveira]

@ayosher sorry for the delay.

Tried that. I still have the same issue. Now with ./do way. I believe I lack some package somewhere - but I have no idea which one...

I still think its a riscv-opcodes problem, because it seems you can not import a file that did not exist in previous version, please try

git submodule update --init --recursive
git submodule update --remote ext/riscv-opcodes

Still, if it does appear the same error, try pip install pyyaml since it the only non-standard python package in the repo.

If it still does not work, please ping me again and so I can take a deeper look trying to figure it out.

[ayosher]

@AFOliveira - now it worked, thank you!

[AFOliveira]

@AFOliveira - now it worked, thank you!

Great!

[AFOliveira]

FYI, this is close to working, but C extensions has shown to be a pain - there's like 20+ operands that need to be further defined by hand, so expect this to take a while to get to 100%.

This is the point we are at:

============================ 64 failed, 1067 passed in 0.81s ============================

[dhower-qc]

@AFOliveira, how does this PR relate to #553 and #526 ?

[AFOliveira]

@AFOliveira, how does this PR relate to #553 and #526 ?

Well, unfortunately, it does not at all. I realized that this task was way to challenging for a "sole" reason: RISC-V opcodes variable naming is some sort of black magic that is almost impossible to solve. This PR serves to show that we are in a comparable state in terms of instructions to what riscv-opcodes has, but I don't think we should integrate it into any workflow, since that there is no clean way reproducing that .json and leaves this code as a really nice spaghetti mess that we don't want to deal with in the future

[dhower-qc]

Ok, should we close this then, or is there a value to keeping it open for the time being?

[AFOliveira]

Closing it for now