refactor(Repository): Add `RepositoryProvenance` attribute #10575

pepper-jk · 2025-07-09T11:03:52Z

This draft includes the first changes for my personal POC for the LocalProvenance Input discussed in #8803.
As it's scope is so small, it might be good first contribution towards that goal.
If not, feel free to let me know. All feedback is welcome.

The Repository data structure seems the easiest place to allow Provenances as Analyzer and Scanner input.
This avoids refactoring the attributes of higher level data structures, such as OrtResult.
At the moment the implementation limits it to RepositoryProvence though, in order to keep its previous behavior. ~~To that end, it also continues to expose the raw VcsInfo of the provenance as its vcs attribute.~~

Signed-off-by: Jens Keim [email protected]

model/src/main/kotlin/Repository.kt

pepper-jk · 2025-09-02T11:13:14Z

Just a rebase onto 66.1.0.

analyzer/src/main/kotlin/Analyzer.kt

model/src/main/kotlin/Repository.kt

evaluator/src/testFixtures/kotlin/TestData.kt

model/src/main/kotlin/Repository.kt

plugins/reporters/fossid/src/test/kotlin/FossIdReporterTest.kt

plugins/reporters/ctrlx/src/funTest/kotlin/CtrlXAutomationReporterFunTest.kt

plugins/reporters/opossum/src/test/kotlin/OpossumReporterTest.kt

pepper-jk · 2025-09-11T13:45:22Z

Okay, according to the tests, I clearly missed something.
Unfortunately, I need to finish up for the week, so I will be back at this on Monday.
If you have any feedback or input on the failing tests in the meantime, it would be greatly appreciated.

codecov · 2025-09-11T13:50:32Z

Codecov Report

❌ Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.54%. Comparing base (783c6d5) to head (c45f7ba).

Files with missing lines	Patch %	Lines
...ands/CreateAnalyzerResultFromPackageListCommand.kt	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10575      +/-   ##
============================================
- Coverage     57.57%   57.54%   -0.03%     
  Complexity     1702     1702              
============================================
  Files           346      346              
  Lines         12829    12833       +4     
  Branches       1212     1212              
============================================
- Hits           7386     7385       -1     
- Misses         4978     4983       +5     
  Partials        465      465

Flag	Coverage Δ
funTest-docker	`71.03% <ø> (+0.04%)`	⬆️
test-ubuntu-24.04	`42.28% <0.00%> (-0.02%)`	⬇️
test-windows-2025	`42.26% <0.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sschuberth · 2025-09-11T18:41:56Z

If you have any feedback or input on the failing tests in the meantime, it would be greatly appreciated.

I think the general problem is that older ORT results that do not contain a RepositoryProvenance still need to be deserializable. You probably need to write a custom deserializer for that...

model/src/main/kotlin/Repository.kt

pepper-jk · 2025-09-16T12:02:43Z

I think the general problem is that older ORT results that do not contain a RepositoryProvenance still need to be deserializable. You probably need to write a custom deserializer for that...

Thanks for the hint, adding the custom Deserializer fixed a lot of tests. Quite some progress today.

Working on fixing the other tests. Will get back to it tomorrow.

pepper-jk · 2025-09-17T15:25:35Z

@sschuberth @mnonnenmacher
Not quite done yet, but feedback on the fixes for the tests so far would be awesome for tomorrow. 😃

sschuberth · 2025-09-17T15:43:29Z

Not quite done yet, but feedback on the fixes for the tests so far would be awesome for tomorrow. 😃

Sorry, I'll not be available tomorrow.

pepper-jk · 2025-09-17T17:00:22Z

Not quite done yet, but feedback on the fixes for the tests so far would be awesome for tomorrow. 😃

Sorry, I'll not be available tomorrow.

Oh, my bad. I forgot. See you next week.

evaluator/src/testFixtures/kotlin/TestData.kt

sschuberth · 2025-09-23T07:30:31Z

@pepper-jk, please rebase to get rid of a test failure.

pepper-jk · 2025-09-23T09:56:03Z

@pepper-jk, please rebase to get rid of a test failure.

I think the failing test is unrelated. The Expat package got a security update on September 16, and Conan is delivering the new package now.

sschuberth · 2025-09-23T10:03:14Z

@pepper-jk, please rebase to get rid of a test failure.

I think the failing test is unrelated. The Expat package got a security update on September 16, and Conan is delivering the new package now.

That's exactly the reason why you should do the rebase 😉 We fixed that already in main.

pepper-jk · 2025-09-23T10:04:38Z

@pepper-jk, please rebase to get rid of a test failure.

I think the failing test is unrelated. The Expat package got a security update on September 16, and Conan is delivering the new package now.

That's exactly the reason why you should do the rebase 😉 We fixed that already in main.

I am already working on the rebase as we speak. I was just checking the tests first to be sure yesterdays changes did not mess anything up.

pepper-jk · 2025-09-23T10:07:35Z

Rebased on 68.2.0 to check in on tests. WIll rebase on main after.

pepper-jk · 2025-09-23T13:10:54Z

Rebased on current main now.

pepper-jk · 2025-09-25T09:16:29Z

As discussed in the community meeting today, the Repository should contain normalized vcsInfo and resolved revisions.

This should solve the remaining two threads. So that is what I will be working on now.

In order to support non-VCS input for ORT, `KnownProvenance`s need to be supported as input for `anaylzer` and `scanner`. The `Repository` data structure appears to be the best place to facilitate this change. This change focusses on adding a `RepositoryProvenance` attribute to `Repository`, which allows to keep previous behavior. For now it also exposes the raw `VcsInfo` of the `provenance` as its `vcs` attribute. Signed-off-by: Jens Keim <[email protected]>

Remove `vcs` and `vcsProcessed` attributes from `Repository`, since the `RepositoryProvenance` `provenance` is now the main attribute. Add `RepositoryDeserializer` to support parsing of legacy `OrtResult` files containing the previous attributes `vcs` and `vcsProcessed`, as well as the new format. It also needs to handle the `nestedRepositories` and `config` attributes for both legacy and recent `OrtResult` files. Signed-off-by: Jens Keim <[email protected]>

Signed-off-by: Jens Keim <[email protected]>

…blank Signed-off-by: Jens Keim <[email protected]>

Signed-off-by: Jens Keim <[email protected]>

Since these files are just parsed as plain text instead of deserialized into ORT data structures, updating them to the syntax appears to be the only way to fix the tests from here on out. Signed-off-by: Jens Keim <[email protected]>

Signed-off-by: Jens Keim <[email protected]>

`Repository.provenance.vcsInfo` would be even better, but KDoc appears to not be able to follow at this attribute depth. Signed-off-by: Jens Keim <[email protected]>

Signed-off-by: Jens Keim <[email protected]>

pepper-jk · 2025-09-25T12:58:11Z

model/src/main/kotlin/Repository.kt

+                // Get the [vcs]'s revision.
+                // Fall back to [vcsProcessed], if [vcs] has empty revision.
+                val resolvedRevision = vcs.revision.ifEmpty {
+                    vcsProcess.revision.ifEmpty {
+                        HashAlgorithm.SHA1.emptyValue
+                    }
+                }


The EvaluatedModelReporterFunTests are currently failing, since the Deserializer can not extract a resolvedRevison from the legacy Repository data structure, as the vcs_processed only references master. See reporter-test-input.yml, which is used as input in the tests.

I assume I would need to iterate over the other nodes in the file and get the resolved_revision from a provenance, e.g. inside the scanner at L542 or the list of provenances at L438.

Since there is a findParent method, I assume it would at least be possible to iterate over other nodes from inside a Deserializer, but I have not confirmed this yet.

@sschuberth @mnonnenmacher Before I continue on this hunch, could you way in on the issue?

pepper-jk · 2025-09-25T15:50:21Z

Rebased onto main aka 69.0.0.

fviernau requested changes Jul 9, 2025

View reviewed changes

model/src/main/kotlin/Repository.kt Outdated Show resolved Hide resolved

pepper-jk force-pushed the repository-provenance-wrapper branch from b608307 to cdc634c Compare July 31, 2025 12:34

pepper-jk force-pushed the repository-provenance-wrapper branch 3 times, most recently from e01ca18 to 1107a05 Compare September 2, 2025 11:09

pepper-jk marked this pull request as ready for review September 4, 2025 08:41

pepper-jk requested a review from a team as a code owner September 4, 2025 08:41

pepper-jk mentioned this pull request Sep 4, 2025

feat(Provenance): Add DirectoryProvenance as a LocalProvenance #9872

Open

mnonnenmacher requested changes Sep 11, 2025

View reviewed changes

pepper-jk force-pushed the repository-provenance-wrapper branch 2 times, most recently from 9ac6ba5 to 101b10c Compare September 11, 2025 13:25

github-advanced-security bot found potential problems Sep 16, 2025

View reviewed changes

model/src/main/kotlin/Repository.kt Fixed Show fixed Hide fixed

model/src/main/kotlin/Repository.kt Fixed Show fixed Hide fixed

pepper-jk force-pushed the repository-provenance-wrapper branch from 9090be5 to cc83150 Compare September 16, 2025 10:50

pepper-jk force-pushed the repository-provenance-wrapper branch 4 times, most recently from 6c12714 to a799cee Compare September 17, 2025 14:25

This comment was marked as outdated.

Sign in to view

pepper-jk requested a review from mnonnenmacher September 18, 2025 10:26

pepper-jk force-pushed the repository-provenance-wrapper branch from f2bdb8c to 5333234 Compare September 18, 2025 11:37

pepper-jk requested review from fviernau and sschuberth September 18, 2025 15:41

github-advanced-security bot found potential problems Sep 22, 2025

View reviewed changes

evaluator/src/testFixtures/kotlin/TestData.kt Fixed Show fixed Hide fixed

evaluator/src/testFixtures/kotlin/TestData.kt Fixed Show fixed Hide fixed

pepper-jk force-pushed the repository-provenance-wrapper branch from d12811f to 601b9ee Compare September 22, 2025 16:35

sschuberth mentioned this pull request Sep 23, 2025

Introduce (reflective) Arb testing with Kotest #10880

Merged

pepper-jk force-pushed the repository-provenance-wrapper branch from 601b9ee to 8615132 Compare September 23, 2025 10:01

pepper-jk force-pushed the repository-provenance-wrapper branch from 8615132 to 07b78f2 Compare September 23, 2025 10:06

pepper-jk force-pushed the repository-provenance-wrapper branch from 07b78f2 to 1230128 Compare September 23, 2025 13:10

pepper-jk force-pushed the repository-provenance-wrapper branch from 62104e4 to 512ec6b Compare September 25, 2025 11:04

pepper-jk added 12 commits September 25, 2025 13:39

fix(OrtResultTest): No more blank revisions

9e4490a

Signed-off-by: Jens Keim <[email protected]>

fix(ScannerIntegrationFunTest): Fallback on emptyValue if revision …

d4a2a91

…blank Signed-off-by: Jens Keim <[email protected]>

fix(ScannerIntegrationFunTest): Add revision to result files

f71005e

Signed-off-by: Jens Keim <[email protected]>

fix(ScannerIntegrationFunTest): Change repo syntax in result files

8237a1f

Signed-off-by: Jens Keim <[email protected]>

fix(OrtResultTest): Avoid setting resolvedRevision after the fact

3a465ca

Signed-off-by: Jens Keim <[email protected]>

fix(KDoc): Reference Repository.provenance now

9590fc0

`Repository.provenance.vcsInfo` would be even better, but KDoc appears to not be able to follow at this attribute depth. Signed-off-by: Jens Keim <[email protected]>

refactor(Analyzer): Always pass vcs.normalize() to Repository

b9c4e68

Signed-off-by: Jens Keim <[email protected]>

fix(EvaluatedModelReporterFunTest): Set correct resolved_revision

dc3e808

Signed-off-by: Jens Keim <[email protected]>

fix(Analyzer): Populate resolvedRevision with Worktree.getRevision()

c45f7ba

Signed-off-by: Jens Keim <[email protected]>

pepper-jk commented Sep 25, 2025

View reviewed changes

pepper-jk force-pushed the repository-provenance-wrapper branch from baab4b2 to c45f7ba Compare September 25, 2025 15:49

pepper-jk mentioned this pull request Oct 22, 2025

Allow "non-repository" input to the analyzer #8803

Open

refactor(Repository): Add RepositoryProvenance attribute #10575

Are you sure you want to change the base?

refactor(Repository): Add RepositoryProvenance attribute #10575

Uh oh!

Conversation

pepper-jk commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pepper-jk commented Sep 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pepper-jk commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sschuberth commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

pepper-jk commented Sep 16, 2025

Uh oh!

pepper-jk commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sschuberth commented Sep 17, 2025

Uh oh!

pepper-jk commented Sep 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

sschuberth commented Sep 23, 2025

Uh oh!

pepper-jk commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sschuberth commented Sep 23, 2025

Uh oh!

pepper-jk commented Sep 23, 2025

Uh oh!

pepper-jk commented Sep 23, 2025

Uh oh!

pepper-jk commented Sep 23, 2025

Uh oh!

pepper-jk commented Sep 25, 2025

Uh oh!

pepper-jk Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

pepper-jk commented Sep 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

refactor(Repository): Add `RepositoryProvenance` attribute #10575

refactor(Repository): Add `RepositoryProvenance` attribute #10575

pepper-jk commented Jul 9, 2025 •

edited

Loading

pepper-jk commented Sep 11, 2025 •

edited

Loading

codecov bot commented Sep 11, 2025 •

edited

Loading

pepper-jk commented Sep 17, 2025 •

edited

Loading

pepper-jk commented Sep 23, 2025 •

edited

Loading