-
Notifications
You must be signed in to change notification settings - Fork 356
Implement ort.yml parsing snippet report generation for SCANOSS
#10004
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
497788c
ff0224c
e5571b4
b8868f5
8bf8a34
1e34588
6fb7e7e
ce607c0
642f189
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| /* | ||
| * Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>) | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * License-Filename: LICENSE | ||
| */ | ||
|
|
||
| plugins { | ||
| // Apply precompiled plugins. | ||
| id("ort-plugin-conventions") | ||
| } | ||
|
|
||
| dependencies { | ||
| api(projects.reporter) | ||
|
|
||
| ksp(projects.reporter) | ||
|
|
||
| implementation(projects.model) | ||
| implementation(projects.plugins.reporters.asciidocReporter) | ||
| implementation(projects.plugins.reporters.freemarkerReporter) | ||
| implementation(projects.utils.commonUtils) | ||
| implementation(projects.utils.ortUtils) | ||
|
|
||
| implementation(libs.kotlinx.coroutines) | ||
|
|
||
| testImplementation(libs.mockk) | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| /* | ||
| * Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>) | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * https://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| * | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * License-Filename: LICENSE | ||
| */ | ||
|
|
||
| package org.ossreviewtoolkit.plugins.reporters.scanoss | ||
|
|
||
| import java.io.File | ||
|
|
||
| import org.ossreviewtoolkit.plugins.api.OrtPlugin | ||
| import org.ossreviewtoolkit.plugins.api.PluginDescriptor | ||
| import org.ossreviewtoolkit.plugins.reporters.asciidoc.AsciiDocTemplateReporterConfig | ||
| import org.ossreviewtoolkit.plugins.reporters.asciidoc.HtmlTemplateReporter | ||
| import org.ossreviewtoolkit.reporter.Reporter | ||
| import org.ossreviewtoolkit.reporter.ReporterFactory | ||
| import org.ossreviewtoolkit.reporter.ReporterInput | ||
|
|
||
| @OrtPlugin( | ||
|
||
| displayName = "SCANOSS Snippet Reporter", | ||
| description = "Generates a detailed report of the SCANOSS snippet findings.", | ||
| factory = ReporterFactory::class | ||
| ) | ||
| class ScanossSnippetReporter(override val descriptor: PluginDescriptor = ScanossSnippetReporterFactory.descriptor) : | ||
|
||
| Reporter by delegateReporter { | ||
| companion object { | ||
| private val delegateReporter = HtmlTemplateReporter( | ||
| ScanossSnippetReporterFactory.descriptor, | ||
| AsciiDocTemplateReporterConfig(templateIds = listOf("scanoss_snippet"), templatePaths = null) | ||
| ) | ||
| } | ||
|
|
||
| override fun generateReport(input: ReporterInput, outputDir: File): List<Result<File>> { | ||
| val hasScanossResults = input.ortResult.scanner?.scanResults?.any { it.scanner.name == "SCANOSS" } == true | ||
| require(hasScanossResults) { "No SCANOSS scan results have been found." } | ||
|
|
||
| return delegateReporter.generateReport(input, outputDir) | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Use Unix line endings for Freemarker templates for consistency across platforms. | ||
| **/*.ftl text eol=lf |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| [#-- | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not the expert here, but do we really need a separate snippet report for SCANOSS? Can't we generalize the existing I'd like to avoid that each new snippet scanner we introduce support for also requires its own report. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The separate scanner reports approach was discussed with Nikola, who preferred keeping them separate due to concerns about diverging functionality. However, you make a valid point. Perhaps having a generic reporter that can adapt to each scanner's particularities would be better? @nnobelis, what are your thoughts on this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. While it is true that the templates are nearly the same, they are still some differences between FossID and ScanOSS. This also make the plugins self-contained, no ? Additionally, The reporters themselves seems to be identical, so maybe they could be merged but the same problem would arise: where the class should be located in ORT ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Well, quite simply, we could have a generic "snippet scanner reporter" in ORT that neither carries "FossID" nor "SCANOSS" in its name, but only mentions the used / configured scanner as part of the generated report. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood. So we can do that for the Reporter class but as said before, I would like to keep two different templates. But this is again increasing the complexity of this PR so would you agree to keep this "shared-reporter" approach in a follow-up PR ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'd agreed to that if it turns out that we really would need too many conditionals in the template. But I'm not yet convinced of that.
Yes, see my wish here to split this PR. |
||
| Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>) | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| https://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
|
|
||
| SPDX-License-Identifier: Apache-2.0 | ||
| License-Filename: LICENSE | ||
| --] | ||
|
|
||
| :publisher: OSS Review Toolkit | ||
| [#assign now = .now] | ||
| :revdate: ${now?date?iso_local} | ||
|
|
||
| :title-page: | ||
| :sectnums: | ||
| :toc: | ||
|
|
||
| = SCANOSS Snippets | ||
| List of all the provenances with their files and snippets. | ||
| [#list ortResult.scanner.scanResults as scanResult] | ||
|
|
||
| [#assign snippetsLimitIssue = helper.getSnippetsLimitIssue()] | ||
|
|
||
| [#if snippetsLimitIssue?has_content] | ||
| [WARNING] | ||
| ==== | ||
| ${snippetsLimitIssue} | ||
| ==== | ||
| [/#if] | ||
|
|
||
| [#if scanResult.provenance.vcsInfo??] | ||
| [#assign url = scanResult.provenance.vcsInfo.url] | ||
| [#else] | ||
| [#assign url = scanResult.provenance.sourceArtifact.url] | ||
| [/#if] | ||
| == Provenance '${url}' | ||
|
|
||
| [#assign summary = scanResult.summary] | ||
|
|
||
| Scan start time : ${summary.startTime} + | ||
| End time : ${summary.startTime} + | ||
| [#if scanResult.provenance.vcsInfo??] | ||
| [#assign gitRepoUrl = url] | ||
| [#assign gitRevision = scanResult.provenance.vcsInfo.revision] | ||
| Git repo URL: ${gitRepoUrl} + | ||
| Git revision: ${gitRevision} | ||
|
|
||
| [#if gitRepoUrl?contains("github.com")] | ||
| [#assign githubBaseURL = '${gitRepoUrl?remove_ending(".git")}/blob/${gitRevision}'] | ||
| [/#if] | ||
| [/#if] | ||
|
|
||
| [#list helper.groupSnippetsByFile(summary.snippetFindings) as filePath, snippetFindings ] | ||
|
|
||
| [#if gitRepoUrl?? && gitRepoUrl?contains("github.com")] | ||
| [#assign localFileURL = '${githubBaseURL}/${filePath}[${filePath}]'] | ||
| [#else] | ||
| [#assign localFileURL = "${filePath}"] | ||
| [/#if] | ||
| [#assign licenses = helper.collectLicenses(snippetFindings)] | ||
|
|
||
| *${localFileURL}* + | ||
| License(s): | ||
| [#list licenses as license] | ||
| ${license}[#sep], | ||
| [/#list] | ||
|
|
||
| [#list helper.groupSnippetsBySourceLines(snippetFindings) as sourceLocation, snippetFinding] | ||
| [#assign snippetCount = snippetFinding.snippets?size] | ||
|
|
||
| [width=100%] | ||
| [cols="1,3,4,1,2"] | ||
| |=== | ||
| | Source Location | pURL | License | Score | Release Date | ||
|
|
||
| .${snippetCount*2}+| | ||
| Partial match + | ||
| ${sourceLocation.startLine?c}-${sourceLocation.endLine?c} | ||
|
|
||
|
|
||
| [#list snippetFinding.snippets as snippet ] | ||
|
|
||
| | ${snippet.purl!""} | ||
| | ${snippet.license!""} | ||
| | ${snippet.score!""} | ||
| | ${snippet.additionalData['release_date']} | ||
|
|
||
| 4+a| | ||
| .Create a snippet choice for this snippet or mark it as false positive | ||
| [%collapsible] | ||
| ==== | ||
| Add the following lines to the *.ort.yml* file. | ||
|
|
||
| To **choose** this snippet: | ||
| [source,yaml] | ||
| -- | ||
| snippet_choices: | ||
| - provenance: | ||
| url: "${scanResult.provenance.vcsInfo.url}" | ||
| choices: | ||
| - given: | ||
| source_location: | ||
| path: "${filePath}" | ||
| start_line: ${snippetFinding.sourceLocation.startLine?c} | ||
| end_line: ${snippetFinding.sourceLocation.endLine?c} | ||
| choice: | ||
| purl: "${snippet.purl!""}" | ||
| reason: "ORIGINAL_FINDING" | ||
| comment: "Explain why this snippet choice was made" | ||
| -- | ||
| Or to mark this location has having ONLY **false positives snippets**: | ||
| [source,yaml] | ||
| -- | ||
| snippet_choices: | ||
| - provenance: | ||
| url: "${scanResult.provenance.vcsInfo.url}" | ||
| choices: | ||
| - given: | ||
| source_location: | ||
| path: "${filePath}" | ||
| start_line: ${snippetFinding.sourceLocation.startLine?c} | ||
| end_line: ${snippetFinding.sourceLocation.endLine?c} | ||
| choice: | ||
| reason: "NO_RELEVANT_FINDING" | ||
| comment: "Explain why this location has only false positives snippets" | ||
| -- | ||
| ==== | ||
| [/#list] | ||
| |=== | ||
| [/#list] | ||
| [/#list] | ||
| [/#list] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| [#ftl] | ||
| [#-- @implicitly included --] | ||
|
|
||
| [#-- @ftlvariable name="projects" type="kotlin.collections.Set<org.ossreviewtoolkit.reporter.utils.FreemarkerTemplateProcessor.PackageModel>" --] | ||
| [#-- @ftlvariable name="packages" type="kotlin.collections.Set<org.ossreviewtoolkit.reporter.utils.FreemarkerTemplateProcessor.PackageModel>" --] | ||
| [#-- @ftlvariable name="ortResult" type="org.ossreviewtoolkit.model.OrtResult" --] | ||
| [#-- @ftlvariable name="licenseTextProvider" type="org.ossreviewtoolkit.reporter.LicenseTextProvider" --] | ||
| [#-- @ftlvariable name="LicenseView" type="org.ossreviewtoolkit.model.licenses.LicenseView" --] | ||
| [#-- @ftlvariable name="helper" type="org.ossreviewtoolkit.plugins.reporters.freemarker.FreemarkerTemplateProcessor.TemplateHelper" --] | ||
| [#-- @ftlvariable name="projectsAsPackages" type="kotlin.collections.Set<org.ossreviewtoolkit.model.Identifier>" --] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we make a cut here and only keep the preceding 5 commits in this PR, and create separate PRs for the next 4 commits? (Probably 2 or 3 other PRs that each focus on report generation or refactoring.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's how I propose to organize them:
PR 1: Migrate to SCANOSS SDK
PR 2: Add Exclusion Pattern & Snippet Choice Support
PR 3: Report Generation
Since these changes build on each other, these PRs would need to be merged sequentially. Would you prefer create all three PRs at once and mark the dependent ones as "Draft" until their predecessors are merged, or should I submit them one at a time as each gets approved?
I can prepare these separate PRs right away once we agree on the approach.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Either way is fine with me. Maybe the latter creates a bit less "clutter" due to open draft PRs.
Great, thanks. Please go ahead as you see fit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First PR is #10265
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@isasmendiagus, is it ok to close this PR already to clean things up, as all its remainders are now included in #10287?