tools: import a conversion tool to and from VuXML #237

khorben · 2024-04-23T15:46:46Z

This is relevant to FreeBSD's ports, and possibly to any other project using the VuXML format in order to track vulnerabilities. (http://vuxml.freebsd.org)

The objective is to help FreeBSD offer security advisories in the OSV format, for ports first but possibly also for base components in the future. The corresponding ecosystem string (most likely FreeBSD) will be requested in a dedicated pull request.

Sponsored by: The FreeBSD Foundation

andrewpollock · 2024-10-16T06:15:37Z

Hi @khorben, this is exciting progress. I can review this from the OSV side of things, is there a VuXML SME who can review that side of things?

Also, if you look at tools/redhat for inspiration, having some test data to validate behaviour is also helpful.

khorben · 2025-05-02T23:59:39Z

Hi Andrew, sorry for the late answer. I suppose by SME you meant "Subject Matter Expert", in which case I think FreeBSD's ports-secteam@ (https://wiki.freebsd.org/PortsSecteam) might be the best point of contact.
I am looking into importing some test data for validation.

andrewpollock · 2025-05-12T06:52:11Z

Would you mind addressing the DCO check failure?

andrewpollock

I'm confused by the UBUNTU-CVE-2025-3454 test files?

I think it'll be a no-op change, but it'd be good to update the schema version in the test OSV records to the current version.

tools/vuxml/README.md

khorben · 2025-05-12T17:51:34Z

Would you mind addressing the DCO check failure?

Thanks for the heads up, I think this should be addressed now.

khorben · 2025-05-12T17:56:04Z

I'm confused by the UBUNTU-CVE-2025-3454 test files?

I have organised the test data as follows:

testdata/VUXML contains the original VUXML files, and their converted output into OSV's JSON
testdata/OSV is the other way around
Should I change or clarify anything in the commits?

This is relevant to FreeBSD's ports. Note that: * There is no official ecosystem for FreeBSD ports yet * A few elements of the VuXML DTD are not covered (e.g., \<uscertsa>, \<uscertta>) * Cancelled VuXML events are not represented, for lack of information (withdrawal date...) * Some versions do not match reality and need a correct increment (\<gt>) Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

XXX this only happens when outputting to the standard output. This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

khorben · 2025-05-12T21:50:08Z

I'm confused by the UBUNTU-CVE-2025-3454 test files?

Oh do you mean that you expected test files about FreeBSD?

First, IIRC VuXML is distribution-agnostic, and not tied to a specific ecosystem.
The FreeBSD project does not provide OSV files (yet); the objective is to allow the project to do so in the future; this is currently being investigated. OTOH, it can already be useful to import OSV data from other projects, and to publish it as VuXML fort the FreeBSD ports after review.

Likewise, it can be useful to convert VuXML data into OSV for the preparation work mentioned above. However, this still requires the corresponding ecosystem for FreeBSD, and choosing its exact name(s) might still be tricky: there is the base system, to distinguish from the third-party ports, and soon also from the packages for the base system.

I think it'll be a no-op change, but it'd be good to update the schema version in the test OSV records to the current version.

I have now updated the schema version in the output to OSV as 1.7.0.
I believe this is now as expected; let me know otherwise.

illuusio · 2025-05-14T07:14:25Z

When turning off DTD validation then it works (dtd_validation=False) with: https://vuxml.freebsd.org/freebsd/vuln.xml

illuusio · 2025-05-14T07:40:47Z

tools/vuxml/convert_vuxml.py

+    if len(args) != 1:
+        return usage()
+
+    parser = etree.XMLParser(dtd_validation=True)


If this is True then it does not work:

File "/somewhere/osv-schema/tools/vuxml/convert_vuxml.py", line 339, in <module> sys.exit(main()) ^^^^^^ File "/somewhere/osv-schema/tools/vuxml/convert_vuxml.py", line 127, in main tree = etree.parse(args[0], parser) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/lxml/etree.pyx", line 3590, in lxml.etree.parse File "src/lxml/parser.pxi", line 1958, in lxml.etree._parseDocument File "src/lxml/parser.pxi", line 1984, in lxml.etree._parseDocumentFromURL File "src/lxml/parser.pxi", line 1887, in lxml.etree._parseDocFromFile File "src/lxml/parser.pxi", line 1200, in lxml.etree._BaseParser._parseDocFromFile File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError File "vuln.xml", line 26 lxml.etree.XMLSyntaxError: failed to load "http:/www.vuxml.org/dtd/vuxml-1/vuxml-11.dtd": Attempt to load network entity, line 26, column 3

illuusio · 2025-05-14T07:55:55Z

I would also add purl for packages when ecosystem have package manager it's nice addition to information

illuusio · 2025-05-14T08:01:51Z

When Output directory is issued then (if I understand correctly) naming should something like: FBSD-2025-05-14.json or FBSD-VUID.json. My humble opinion is that they should be arranged by package name or yearly so there should not be so much in one dir.

illuusio · 2025-05-14T08:27:00Z

It seems that <affects> is right next to <topic> in FreeBSD VuXML-file. It shouldn't affect anything but if parser is very bad then it does not work.

illuusio · 2025-05-14T08:37:43Z

tools/vuxml/convert_osv.py

@@ -0,0 +1,271 @@
+#!/usr/bin/env python
+#


Just suggesting to have SPDX-header for license also

Thank you for the suggestion, I have now added the SPDX headers.

SPDX header seems to be correct

Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

oliverchang

thank you very much for working on this, and for adding the tests! (and apologies for the delay in reviewing)

oliverchang · 2025-06-03T09:14:36Z

tools/vuxml/testdata/VUXML/CVE-2025-29087.json

@@ -0,0 +1,42 @@
+{
+    "schema_version": "1.7.0",
+    "id": "409206f6-25e6-11f0-9360-b42e991fc52e",


this needs to have a FREEBSD- or some other prefix so it's clear it comes from the FreeBSD database.

Please also define the chosen prefix in https://ossf.github.io/osv-schema/#id-modified-fields

oliverchang · 2025-06-03T09:15:10Z

tools/vuxml/testdata/VUXML/CVE-2025-29087.json

+    "affected": [
+        {
+            "package": {
+                "ecosystem": "FreeBSD:ports",


In a separate PR, could you please add a ecosystem definition to https://ossf.github.io/osv-schema/#affectedpackage-field ?

oliverchang · 2025-06-03T09:16:18Z

tools/vuxml/testdata/VUXML/CVE-2025-29087.json

+            },
+            "ranges": [
+                {
+                    "type": "SEMVER",


nit: does FreeBSD follow strict Semver 2.0 per https://semver.org/ and all its ordering rules?

If not, "ECOSYSTEM" would be the more accurate type.

khorben force-pushed the khorben/vuxml branch from 60797b2 to b7ea020 Compare October 15, 2024 15:05

khorben marked this pull request as ready for review October 15, 2024 15:06

khorben force-pushed the khorben/vuxml branch 3 times, most recently from 4bd95a4 to 8614a9b Compare May 3, 2025 01:01

andrewpollock reviewed May 12, 2025

View reviewed changes

tools/vuxml/README.md Show resolved Hide resolved

khorben force-pushed the khorben/vuxml branch 2 times, most recently from 3e0b468 to d3c580a Compare May 12, 2025 17:50

khorben force-pushed the khorben/vuxml branch from d3c580a to 18503b2 Compare May 12, 2025 21:36

khorben added 4 commits May 12, 2025 23:39

tools: import a conversion tool to vuxml

368b217

This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

tools: output a JSON object for single entries

c4bf11c

XXX this only happens when outputting to the standard output. This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

tools: import test data for the vuxml conversion tool

44d9763

This is relevant to FreeBSD's ports. Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

khorben force-pushed the khorben/vuxml branch from 18503b2 to 44d9763 Compare May 12, 2025 21:39

illuusio reviewed May 14, 2025

View reviewed changes

tools: add SPDX headers to the vuxml conversion tools

62ccc24

Sponsored by: The FreeBSD Foundation Signed-off-by: Pierre Pronchery <[email protected]>

oliverchang reviewed Jun 3, 2025

View reviewed changes

tools: import a conversion tool to and from VuXML #237

Are you sure you want to change the base?

tools: import a conversion tool to and from VuXML #237

Uh oh!

Conversation

khorben commented Apr 23, 2024

Uh oh!

andrewpollock commented Oct 16, 2024

Uh oh!

khorben commented May 2, 2025

Uh oh!

andrewpollock commented May 12, 2025

Uh oh!

andrewpollock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khorben commented May 12, 2025

Uh oh!

khorben commented May 12, 2025

Uh oh!

khorben commented May 12, 2025

Uh oh!

illuusio commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illuusio May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

illuusio commented May 14, 2025

Uh oh!

illuusio commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illuusio commented May 14, 2025

Uh oh!

illuusio May 14, 2025

Choose a reason for hiding this comment

Uh oh!

khorben May 29, 2025

Choose a reason for hiding this comment

Uh oh!

illuusio Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

oliverchang left a comment

Choose a reason for hiding this comment

Uh oh!

oliverchang Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

oliverchang Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

oliverchang Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

illuusio commented May 14, 2025 •

edited

Loading

illuusio May 14, 2025 •

edited

Loading

illuusio commented May 14, 2025 •

edited

Loading