Pipe commands into OutputFilter

[tasgomes]

For each project, we use a Makefile to manage our (VUnit) simulation and the Vivado project (create/open, build, export, etc.), among other things.

We already have in place some functionality to achieve some of the functionality provided by the OutputFilter. For example, we use pipe commands to pass the output of our console to the sed command, which then colorizes warnings, critical warnings, errors and failures. This is quite useful to identify dangerous warnings in our local console or CI/CD console.

We also use pipe commands to write the console output into log files, including filtered log files that contain only warnings using the tee command.

I see the OutputFilter as something that provides some other extra features and probably more flexibility for expansion.

Do you think I could use this approach to pipe the output of our Makefile commands into the OutputFilter? Is this something already supported or planned to be supported?

[Paebbels]

We already have in place some functionality to achieve some of the functionality provided by the OutputFilter. For example, we use pipe commands to pass the output of our console to the sed command, which then colorizes warnings, critical warnings, errors and failures. This is quite useful to identify dangerous warnings in our local console or CI/CD console.

I have a similar solution at my employer. Here we use a Bash script that receives piped data from Vivado or Riviera-PRO and runs in a while-loop reading lines from STDIN. Internally, it has string matches or some regexps to identify VivadoMessages and or critical lines. The Bash script opens multiple file handles and writes e.g. all INFO messages into one file or all messages of ID 8-7129 into another file, while returning a colored output to STDOUT. In addition, on a critical line, it overrules the return code of Vivado, Riviera-PRO, etc. for CI.

All this is the goal of pyEDAA.OutputFilter.

Usecases I see:

• Embedding it into own Python scripting.
  This would allow projects to share filters and parsing features -- no reinventions of the wheel and combined forces in handling log format variants.
• Using pyEDAA-OutputFilter's CLI command.
  Currently, the CLI is more like a demonstrator, but it could be extended by the usecase you described.
  CLI arguments can be added to select what should be written into separate files (like a standalone CriticalWarning.log file).
• Use in combination with pyEDAA.CLITool and pyEDAA.CLISetup.
  This would use the capability of CLITool to start Vivado using correct CLI parameters in a Python subprocess. The output is then filtered and colored.

---

Do you think I could use this approach to pipe the output of our Makefile commands into the OutputFilter? Is this something already supported or planned to be supported?

Let me see, if I can add another mode to the CLI program. Instead of reading from an existing file, to read from STDIN. Then we have a starting point for your request and we can add further extensions to it.

---

For each project, we use a Makefile to manage our (VUnit) simulation and the Vivado project (create/open, build, export, etc.), among other things.

Do you know of pyEDAA.ProjectModel for a data model to handle source files and filesets for EDA projects?

[tasgomes]

Having the CLI program mode would be great and could open possibilities for us to try it out and possible migrate this part of our flow to the OutputFilter. This would open doors not only for coloring and log files but for more features that the OutputFilter offers.

Do you know of pyEDAA.ProjectModel for a data model to handle source files and filesets for EDA projects?

No, I wasn't aware of it. However, at the moment, we are quite happy with our Makefile-flow.

• It manages our simulations
  • Create (co-)sim models
  • Compile vendor libs
  • Run simulation
    • A specific test case or the whole regression test)
    • Supports GUI mode or console mode
  • Open coverage analysis in GUI
  • Supports any simulator (e.g. Modelsim, Questa, GHDL, NVC, etc.)
• It manages our Vivado projects
  • Create project
  • Build project
  • Export binaries
  • Save the project in TCLs (e.g. it saves a TCL per BD hierarchy, etc).
  • Open Vivado project in GUI mode
• It also tracks that the correct tool versions are used in a particular project so that any developer uses the same tools and their versions, in order to get consistent results.
• It can easily be used by the developer during development or also by CI/CD tools to generate test and official builds

[Paebbels]

I pushed a first commit (90567f4) supporting pyEDAA-OutputFilter vivado --stdin --colored instead of --file=logfile.vds|vdi.

The coloring is currently a bit broken, because I'm switching from hard coded colors in an if-elif structure to a dictionary with color codes. But it should be sufficient to run a first experiment using the latest dev-branch.

If you're on Windows, you can run run.ps1 -build to build the wheel and run.ps1 -install install it using administrator rights as system package. Otherwise use your normal installation process for wheel packages. The wheel is in ./dist

What outputs do you need from a CLI tool?

Some ideas from my side:

• --logfile=vivado.log - write a copy of stdin to this logfile without colors (similar to tee)
• --json=lines.jsonl - write the classified data in JSONlines format
• --warningfile=warnings.log - write all warning to this file
• --criticalfile=criticalwarnings.log - write all critical warning to this file
• --errorfile=errors.log - write all errors to this file
• --messages=8-7129:message.8-7129.log;17-14:message.17-14.log - write messages of these IDs to a separate file.
• --input=- - maybe replacing --stdin with value - on the input file (widely used by CLI programs)
• --output[=colored] - if just --output, write to stdout, if --output=colored write colored output.
• -q|--quiet - will not print a program header
• --summary[=summary.json] - writing the extracted data to stdout or into a JSON file.

Other ideas I have are a --policy file and a --colors file or so. Maybe just one YAML to configure it all.

[tasgomes]

@Paebbels I was able to install it, after installing all the required python packages, and run some tests . The coloring seems indeed broken. At least the string "WARNING:" has no color.

From our side, the following would be good to have in order to achieve the same functionality that we already have with our custom solution:

• possibility to define the name of the log file and its location, e.g. --logfile=log/build_project.log
• possibility to write a log with filtered messages, e.g. containing only warnings, critical warnings and errors. Maybe this can be implemented like log levels in standard log libraries where you can define the filtering threshold, so that for example all levels above and including the log-level "warning" gets written into that file.
• possibility to set the color for each kind of message (e.g. error, warning, critical warning, failure, etc.) - this is not critical but people do have different color preferences.
• I like the option -q|--quiet that will not print a program header.

I think we these 4 options I can replace our current solution with this OutputFilter and the user would not noticed 😄

Two more topics I would like to point out:

1. Sometimes when using the OutputFilter I only get the output of the console after the whole process is finished. It looks like some buffering of the console is going on. In python this can generally be fixed using python -u <script>. Can you reproduce this?
2. Are there other ways of installing this tool? Some way that is similar no matter we are running Linux or Windows (using a bash like git-bash). Currently, my custom solution is based on a few lines in the Makefile... while this one requires installation of an extra tool, so it would help if we can simplify it as much as possible. For example, making sure that the same command can install it no matter if we are on Linux or Windows.

[Paebbels]

The coloring seems indeed broken.

Yes, that's what I meant by I'm rewriting it to use color tables / dictionaries :)

people do have different color preferences.

When I have a default color table, I want to add the ability of loading a settings file (maybe YAML or JSON), which will have one section / key for the coloring. If missing it will use default coloring, if present it will use the user-defined colors.

Maybe this can be implemented like log levels in standard log libraries where you can define the filtering threshold, so that for example all levels above and including the log-level "warning" gets written into that file.

Do you think it's helpful to filter like warning+? Especially Vivado has so many critical messages hidden in lower levels...

So you would prefer an output file, where warning and critical warning can be mixed, right? (Of cause I'll preserve the order of messages.)

I like the option -q|--quiet that will not print a program header.

I think in case of --stdin, this option will be active by default.

---

Sometimes when using the OutputFilter I only get the output of the console after the whole process is finished.

Can you please describe your setup. Windows or Linux? PowerShell or Bash? Was it directly piped from Vivado? Can you provide the command line how you called Vivado and the script?

I didn't test it with Vivado yet. I used cat on a vds file for testing.

I know Vivado has some buffering problems, because on CI server is processes already several hundred lines until the CI log updates, then it's realtime...

-u     : force the stdout and stderr streams to be unbuffered;
         this option has no effect on stdin; also PYTHONUNBUFFERED=x

I didn't know about -u on the Python executable. I usually run Python with PYTHONUNBUFFERED in PyCharm, so stdout and stderr stay aligned. Otherwise error traces get mixed into outputs :(.
I never noticed a big impact on normal execution. I think unbuffered shouldn't be a problem for this usecase, as Vivado spits out a few lines every few seconds.

---

Are there other ways of installing this tool?

The described way was for using the dev branch and building a local wheel + installing it. Normal releases like latest v0.3.0 are published at PyPI. When I fixed the color codes, also v0.4.0 will be at PyPI. Then you have a normal installation flow using PIP.
See https://edaa-org.github.io/pyEDAA.OutputFilter/Installation.html for the normal installation flow.

Does it fulfill you needs?

[Paebbels]

I had a second thought about -u. How to apply it when the CLI is used via pyEDAA-OutputFilter. There is no python program involved.

[tasgomes]

Do you think it's helpful to filter like warning+? Especially Vivado has so many critical messages hidden in lower levels...

So you would prefer an output file, where warning and critical warning can be mixed, right? (Of cause I'll preserve the order of messages.)

Perhaps you can make this configurable. Some may want warnings separated from critical warnings, while others like our case, we want to write everything that is warnings, critical warnings and errors in a file.

Can you please describe your setup. Windows or Linux? PowerShell or Bash? Was it directly piped from Vivado? Can you provide the command line how you called Vivado and the script?

Our setup can run on Linux or on Windows. On Windows we run on bash (git-bash). The command that our Makefile executes is:

cd C:/_build/etlw_001/fw/par && cmd //c "C:/Xilinx/Vivado/2024.2/settings64.bat && vivado" -mode batch -nojournal -nolog -notrace -source C:/_build/etlw_001/fw/par/scripts/save_project.tcl -tclargs --project_dir C:/_build/etlw_001/fw/par/project --project_name project --write_bd_layout 1 --split_create_project 1 |& C:/Python/Python313/Scripts/pyedaa-outputfilter.exe vivado --stdin --colored ; exit "${PIPESTATUS[0]}"

With this command, the console only prints the output at the end of the Vivado process. While with my original command, I can see the console output as the Vivado process goes along. So it seems that when using the OutputFilter the console gets buffered somewhere.

I created a simple test_buffering.tcl script that you can use to reproduce this:

for {set i 1} {$i <= 5} {incr i} {
    puts "Line $i"
    after 1000  ;# Wait 1 second
}

When using cmd //c "C:/Xilinx/Vivado/2024.2/settings64.bat && vivado" -mode batch -nojournal -nolog -notrace -source C:/_build/etlw_001/fw/par/scripts/test_buffering.tcl, I can see each line being printed after 1 second.

When using cmd //c "C:/Xilinx/Vivado/2024.2/settings64.bat && vivado" -mode batch -nojournal -nolog -notrace -source C:/_build/etlw_001/fw/par/scripts/test_buffering.tcl |& C:/Python/Python313/Scripts/pyedaa-outputfilter.exe vivado --stdin --colored, the output is buffered and I get the whole output in one go only after 5/6 seconds.

The described way was for using the dev branch and building a local wheel + installing it. Normal releases like latest v0.3.0 are published at PyPI. When I fixed the color codes, also v0.4.0 will be at PyPI. Then you have a normal installation flow using PIP.
See https://edaa-org.github.io/pyEDAA.OutputFilter/Installation.html for the normal installation flow.

Does it fulfill you needs?

Yes, installing it with pip is perfect.

[Paebbels]

All code for v0.4.0 is complete. I just need to fix one testcase (or convince my CI pipeline to trust me ... ). Changes are described in #25.

Writing outputs to separate files as discussed above, will be part of upcoming v0.5.0.

[Paebbels]

@tasgomes v0.4.0 is currently running its release pipeline. This will allow reading from STDIN via --stdin or --file=-

[tasgomes]

Thanks, looking forward for the v0.5.0.

[tasgomes]

@Paebbels Any estimation when v0.5.0 will be ready?