Duplicated entries in generated SPDX

[dgutson]

I found duplicates in the generated SPDX:

- SPDXID: "SPDXRef-Package-NPM-webassemblyjs-wast-parser-1.9.0"
  copyrightText: "Copyright (c) 2008 Fair Oaks Labs, Inc.\nCopyright (c) 2017 Mauro\
    \ Bringolf\nCopyright (c) 2018 Sven Sauleau <[email protected]>\nCopyright (c)\
    \ Facebook, Inc. and its affiliates\nCopyright 2012 The Obvious Corporation\n\
    Copyright 2012 The Obvious Corporation. http://obvious.com/"
  downloadLocation: "NONE"
  externalRefs:
  - referenceCategory: "PACKAGE_MANAGER"
    referenceType: "purl"
    referenceLocator: "pkg:npm/%40webassemblyjs/[email protected]"
  filesAnalyzed: false
  homepage: "https://github.com/xtuc/webassemblyjs#readme"
  licenseConcluded: "NOASSERTION"
  licenseDeclared: "MIT"
  name: "wast-parser"
  summary: "WebAssembly text format parser"
  versionInfo: "1.9.0"
- SPDXID: "SPDXRef-Package-NPM-webassemblyjs-wast-parser-1.9.0-vcs"
  copyrightText: "Copyright (c) 2008 Fair Oaks Labs, Inc.\nCopyright (c) 2017 Mauro\
    \ Bringolf\nCopyright (c) 2018 Sven Sauleau <[email protected]>\nCopyright (c)\
    \ Facebook, Inc. and its affiliates\nCopyright 2012 The Obvious Corporation\n\
    Copyright 2012 The Obvious Corporation. http://obvious.com/"
  downloadLocation: "git+https://github.com/xtuc/webassemblyjs.git@0440b420888c1f7701eb9762ec657775506b87d8"
  externalRefs:
  - referenceCategory: "PACKAGE_MANAGER"
    referenceType: "purl"
    referenceLocator: "pkg:npm/%40webassemblyjs/[email protected]"
  filesAnalyzed: false
  homepage: "https://github.com/xtuc/webassemblyjs#readme"
  licenseConcluded: "NOASSERTION"
  licenseDeclared: "NOASSERTION"
  name: "wast-parser"
  summary: "WebAssembly text format parser"
  versionInfo: "1.9.0"

Same version of webassemblyjs is duplicated (one with licenseDeclared determined, the other with NOASSERTION), I assume that it is because it was depended from two packages.

Is this expected? Shouldn't be only one entry? (with the same licensing data?). I see that one SPDXID shows that it comes from vcs and the other does not, but this causes us to post process this SPDX file and de-duplicate these entries.

[sschuberth]

Same version of webassemblyjs is duplicated

First of all, strictly speaking there is no duplicate as the SPDXIDs differ, see the "-vcs" suffix.

Is this expected?

Yes.

Shouldn't be only one entry?

No.

I see that one SPDXID shows that it comes from vcs and the other does not, but this causes us to post process this SPDX file and de-duplicate these entries.

There's an inherent problem with SPDX in that it cannot handle multiple different source code locations for a single package (like ORT can). For example for Maven artifacts, there's no (sane) way for a single SPDX package to say where it's sources artifact and VCS location are, if both exist. Our work-around for this limitation in SPDX is to create dedicated SPDX packages per source code location, and link these via the relationships section. @fviernau probably could go into the details.

[dgutson]

@sschuberth imagine this situation: we have to present the SPDX of our SBOM to an auditing authority or to the legal department of a potential partnet or customer. How much do you think they care about the source (-vcs vs no -vcs) when they see two "pseudo identical" entries (at their eyes, all they care is the package name, version, and license but it doesn't match).
Your explanation is a justification from the implementation point of view that for me the user is still a problem to solve. At tue end of the day, I have to ignore how the spdx was generated (whether different sources or not) or the limitations of the SPDX format: I need an SPDX with no "non-strictly" duplicates.
If that takes an extra flag that tells the SPDX exporter "hey sodx exporter, please generate (merge into) only one entry for items of the same source" and it is not a priority for you, then we will implement it and please help us to get it merged.

[sschuberth]

How much do you think they care about the source (-vcs vs no -vcs) when they see two "pseudo identical" entries (at their eyes, all they care is the package name, version, and license but it doesn't match).

I don't know, and I don't want to speculate.

If that takes an extra flag that tells the SPDX exporter "hey sodx exporter, please generate (merge into) only one entry for items of the same source" and it is not a priority for you, then we will implement it and please help us to get it merged.

Please first get in touch with @fviernau to discuss whether he'd be fine with having such a feature flag at all.

[tsteenbe]

@dgutson This is actually the correct way in SPDX to implement the reality in code. You have a project that includes wast-parser 1.9.0 release artifact (SPDX Package 1) that based upon code repository github.com/xtuc/webassemblyjs.git (SPDX Package 2). We will soon begin to implement file findings in SPDX output which why there are 2 packages in ORT's SPDX output - the file findings will need to be connected to the package for the code repository or source artifact not the release artifact.

If we upgrade SPDX generation to output version 2.3 instead of 2.2 we should be able to create some more compact SPDX files as a lot of fields have become options.

See some for implements here - noticed also the purl on SPDXRef-Package-NPM-webassemblyjs-wast-parser-1.9.0-vcs is wrong points to release artifact instead of the source repository.

[tsteenbe]

If that takes an extra flag that tells the SPDX exporter "hey sodx exporter, please generate (merge into) only one entry for items of the same source" and it is not a priority for you, then we will implement it and please help us to get it merged.

This is the wrong approach - we need optional flags for SPDX reporter either

• Only include information about dependency as used by the project (binary artifact)
• Only include information about dependency as used by the project (binary Artifact) and summary of license findings in the code repository / source code artifact that was scanned without details of the files scanned
• Include everything ORT has found

And we need a option to include Advisor results as well...

[dgutson]

@tsteenbe ok, what's the direction then, what's the plan to follow?
I will soon get new interns.

[tsteenbe]

@dgutson Let's talk to see how we can align, please ping me over Slack or drop me an email.

[dgutson]

@dgutson Let's talk to see how we can align, please ping me over Slack or drop me an email.

Check spam?