CycloneDx param generates an empty file

[Gator8]

Attempting to run scancode 31.2.4 with the --cyclonedx or --cyclonedx-xml param (also including --package). Looking to funnel the xml into Dependency-Track

Description

The scan runs, but the file generated is empty of substance. There is no error generated as with previous versions, but the file output is minimal. If I specify the --json-pp option, then the file is populated.

How To Reproduce

Ran the command in several ways in an attempt to narrow down the issue. I stripped out all other options and focused on what would work to have the output file populated.

scancode -clpi -n 8 --package --cyclonedx-xml SCBom.xml .

Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights, packages with 8 process(es)...
[####################] 13174
Scanning done.
Summary: info, licenses, copyrights, packages with 8 process(es)
Errors count: 0
Scan Speed: 3.92 files/sec. 661.24 KB/sec.
Initial counts: 7548 resource(s): 6587 file(s) and 961 directorie(s)
Final counts: 7548 resource(s): 6587 file(s) and 961 directorie(s) for 1.06 GB
Timings:
scan_start: 2023-02-13T155705.253586
scan_end: 2023-02-13T162509.276539
setup_scan:licenses: 1.54s
setup: 1.55s
inventory: 1.86s
scan:packages: 0.30s
scan: 1679.32s
total: 1684.30s
Removing temporary files...done.

scancode -clpi -n 8 --package --cyclonedx SCBom.json .

Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights, packages with 8 process(es)...
[####################] 13172
Scanning done.
Summary: info, licenses, copyrights, packages with 8 process(es)
Errors count: 0
Scan Speed: 4.17 files/sec. 702.55 KB/sec.
Initial counts: 7547 resource(s): 6586 file(s) and 961 directorie(s)
Final counts: 7547 resource(s): 6586 file(s) and 961 directorie(s) for 1.06 GB
Timings:
scan_start: 2023-02-13T153015.612640
scan_end: 2023-02-13T155640.850806
setup_scan:licenses: 1.54s
setup: 1.55s
inventory: 1.87s
scan:packages: 0.28s
scan: 1580.58s
total: 1585.50s
Removing temporary files...done.

System configuration

The project this is being run on is C++/C#. We also have several 3rd party libraries in the solution.

SCBom.zip

[fdx0]

I'm running into this problem as well.

[fdx0]

I'm also seeing this

▶ scancode -p --json-pp output.json --cyclonedx cyclonedx.json test_project
Setup plugins...
Collect file inventory...
Scan files for: packages with 1 process(es)...
[####################] 1206
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/formattedcode/output_cyclonedx.py", line 794, in process_codebase
    bom = CycloneDxBom.from_codebase(codebase)
  File "/usr/local/lib/python3.9/site-packages/formattedcode/output_cyclonedx.py", line 681, in from_codebase
    components = list(CycloneDxComponent.from_packages(packages))
  File "/usr/local/lib/python3.9/site-packages/formattedcode/output_cyclonedx.py", line 339, in from_packages
    base_component.merge(other_component)
  File "/usr/local/lib/python3.9/site-packages/formattedcode/output_cyclonedx.py", line 375, in merge
    merge_lists(self.externalReferences, other.externalReferences)
  File "/usr/local/lib/python3.9/site-packages/formattedcode/output_cyclonedx.py", line 431, in merge_lists
    seen = set(x)
TypeError: unhashable type: 'CycloneDxExternalRef'

Scanning done.
Summary:        packages with 1 process(es)
Errors count:   0
Scan Speed:     65.22 files/sec.
Initial counts: 721 resource(s): 603 file(s) and 118 directorie(s)
Final counts:   721 resource(s): 603 file(s) and 118 directorie(s)
Timings:
  scan_start: 2023-03-14T225115.127298
  scan_end:   2023-03-14T225127.080511
  setup_scan:licenses: 2.08s
  setup: 2.08s
  inventory: 0.53s
  scan: 9.25s
  total: 12.05s
Removing temporary files...done.

▶ scancode --version
ScanCode version: 31.2.4
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.18

[AyanSinhaMahapatra]

@fdx0 the unhashable type bug is fixed by #3189 and available in develop/latest RC releases like v32.0.0rc2.

Can you give us more details on the empty cyclonedx output, if possible, what were you scanning, if the --json output has packages detected?

[Gator8]

We are scanning all of the files that go into our installer for the product.
Our commandline looks like this: c:\PROJECTS\Z_TOOLS\scancode-toolkit-v31.2.4\scancode.bat -clpi -n 8 --verbose --strip-root --verbose --package --cyclonedx result.json --json-pp resultpp.json ./installfiles
resultpp.json ends up being 970k in size. result.json ends up being 2k in size.
The components and dependencies section just never get filled out in 'result.json'
The packages section of the resultpp.json is also empty

[AyanSinhaMahapatra]

@Gator8

The packages section of the resultpp.json is also empty

This is why the cyclonedx document is also empty as we use these top-level packages to populate the cyclonedx document.
So the issue might be a bug where we are wrongly not detecting packages, but we need to know more about the files/package scanned here to debug this more.

[Jasper-Ben]

Just ran into this problem as well (same use-case of importing into dependency track).
@AyanSinhaMahapatra is there are way we can help to get this fixed?

[Jasper-Ben]

recipe-lua-native.tar.gz
I uploaded an example file for you to check @AyanSinhaMahapatra.

This is why the cyclonedx document is also empty as we use these top-level packages to populate the cyclonedx document.

Honestly, I am a bit confused by this statement, mind you I have no idea how the cyclonedx format compares to the json format. I would naively expect the cyclonedx output to work exactly the same as the json format, so why does the cyclonedx format require a top-level package to be detected and json output not?

Maybe for additional context: The packages I am trying to scan are source dumps of an embedded linux created with yocto.

[mjherzog]

@Jasper-Ben CycloneDX is not really designed for communicating about the content of a source archive - it is designed for "higher-level components as describe at https://cyclonedx.org/specification/overview/#components:
"Components describe the complete inventory of first-party and third-party components. Component identity can be represented as:

• Coordinates (group, name, version)
• Package URL
• Common Platform Enumeration (CPE)
• SWID (ISO/IEC 19770-2:2015)
• Cryptographic hash functions (SHA-1, SHA-2, SHA-3, BLAKE2b, BLAKE3)

CycloneDX can represent applications, frameworks, libraries, containers, operating systems, devices, firmware, files, along with the manufacturer information, license and copyright details, and complete pedigree and provenance for every component."

ScanCode uses Package URL (purl) as the Component identifier. Purls normally describe packages created with a package manager like maven or npm. You could create a "generic" purl (https://github.com/package-url/purl-spec) for recipe-lua-native.tar.gz as a package, but purls do not describe the contents of a package (i.e. folder and files).

[Jasper-Ben]

@mjherzog Thank you for the explanation! So if I understood you correctly, cyclonedx isn't really that useful for exchanging detailed information about individual files in source code? Bummer.

A bit OT: Right now we are really struggling to find the right tooling. We really hoped that scancode-toolkit+dependencytrack would bring us the perfect tooling combination, but dependencytrack really seems to be primary focused on vulnerability management.

We want to make legal happy, who insist on a full scan of the entire embedded linux system on a file level. While it is possible to generate a JSON file from all the source code packages using scancode-toolkit, you can imagine the sheer amount of files we need to go through for a whole operating system. And even if we managed once, we need to repeat the whole process from the beginning for the next release.

So we really need a tool that allows for:

1. distributing clearing workload over multiple users
2. can clear files in bulk
3. can reuse clearing status for unchanged files from previous releases

So far, the closest we have come is Fossology (no hate, it is FOSS, provided free-of-charge), however it

a) doesn't scale well for our amount of source code
b) lost our source code collection during an upgrade (with our amount of data an absolute no-go)
c) lacks certain features we deem necessary (i.e. lack of OpenID/LDAP support, API as first-class citizen, ...)

I hope this isn't too much to ask, but maybe you have a suggestion on this? Ideally FOSS, so that we have control over the source code if necessary, but if you know commercial products that work well for our use-case that would also be fine.

[mjherzog]

@Jasper-Ben Siemens has developed a taxonomy for CycloneDX extensions to handle files - see https://github.com/siemens/cyclonedx-property-taxonomy, but I do not know of any public tools that support this. If you are more concerned about licensing than vulnerabilities you may want to consider using the SPDX SBOM format instead of CycloneDX.

Some years ago we helped the Kernel team use ScanCode to analyze the base kernel so that they could add SPDX license identifiers in all source files so that could help, but performing "full scan of the entire embedded linux system on a file level" for each release is a sisyphean task however you do it.

We would be happy to schedule a meeting to discuss the big picture directly. You can reach me at mherzog@nexb.com.

[Jasper-Ben]

@mjherzog Thank you, we will gladly take you up on that offer!

[shusriva]

@Gator8

The packages section of the resultpp.json is also empty

This is why the cyclonedx document is also empty as we use these top-level packages to populate the cyclonedx document. So the issue might be a bug where we are wrongly not detecting packages, but we need to know more about the files/package scanned here to debug this more.

@AyanSinhaMahapatra I am also scanning kong source code which is built on top of lua. result of cyclonedx is very minimal (only around 3 components) whereas jsonpp output contains list of all elements.

I thought of using scancode to generate the sbom for kong source code but it just didn't work as expected.

Any suggestion

[pombredanne]

@shusriva the thing is that CycloneDX does not promote much reporting file details but rather the packages (aka. components)

The way out may include:

• We have an AboutCode CDX extension for this registered at https://github.com/CycloneDX/cyclonedx-property-taxonomy and https://github.com/nexB/aboutcode-cyclonedx-taxonomy#readme and this could treat each file as a CDX component?

• Alternatively we are experimenting in ScanCode.io to create packages from files not in a package and that would share the same licensing.

• Note also that we do not support (yet) Lua rocks as in https://github.com/Kong/kong/blob/master/kong-3.5.0-0.rockspec so we mostly pick other packages like rust and Go

• Another thing is whether to include package details for deps merely listed in lockfiles.