Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions buildSrc/src/main/kotlin/LicenseUtils.kt
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ object CopyrightableFiles {
"plugins/reporters/asciidoc/src/main/resources/pdf-theme/pdf-theme.yml",
"plugins/reporters/asciidoc/src/main/resources/templates/freemarker_implicit.ftl",
"plugins/reporters/fossid/src/main/resources/templates/freemarker_implicit.ftl",
"plugins/reporters/scanoss/src/main/resources/templates/freemarker_implicit.ftl",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make a cut here and only keep the preceding 5 commits in this PR, and create separate PRs for the next 4 commits? (Probably 2 or 3 other PRs that each focus on report generation or refactoring.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's how I propose to organize them:

PR 1: Migrate to SCANOSS SDK

  • refactor(scanoss): Remove path anonymization from SCANOSS implementation
  • refactor(scanoss): Replace direct API calls with SCANOSS SDK
  • refactor(scanoss): Set SCANOSS matcher property to null

PR 2: Add Exclusion Pattern & Snippet Choice Support

  • feat(scanoss): Add exclusion pattern support to SCANOSS
  • feat(scanoss): Add snippet choice parsing for scan results

PR 3: Report Generation

  • feat(scanoss): Add snippet report generation
  • feat(scanoss): Add release date to snippet findings
  • refactor(scanoss): ScanOssResultParser to improve snippets findings

Since these changes build on each other, these PRs would need to be merged sequentially. Would you prefer create all three PRs at once and mark the dependent ones as "Draft" until their predecessors are merged, or should I submit them one at a time as each gets approved?

I can prepare these separate PRs right away once we agree on the approach.

Copy link
Member

@sschuberth sschuberth May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you prefer create all three PRs at once and mark the dependent ones as "Draft" until their predecessors are merged, or should I submit them one at a time as each gets approved?

Either way is fine with me. Maybe the latter creates a bit less "clutter" due to open draft PRs.

I can prepare these separate PRs right away once we agree on the approach.

Great, thanks. Please go ahead as you see fit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First PR is #10265

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isasmendiagus, is it ok to close this PR already to clean things up, as all its remainders are now included in #10287?

"plugins/reporters/freemarker/src/main/resources/templates/freemarker_implicit.ftl",
"plugins/reporters/static-html/src/main/resources/prismjs/",
"plugins/reporters/web-app-template/yarn.lock",
Expand Down
2 changes: 1 addition & 1 deletion integrations/completions/ort-completion.fish
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ complete -c ort -f -n __fish_use_subcommand -a report -d 'Present Analyzer, Scan
## Options for report
complete -c ort -n "__fish_seen_subcommand_from report" -l ort-file -s i -r -F -d 'The ORT result file to use.'
complete -c ort -n "__fish_seen_subcommand_from report" -l output-dir -s o -r -F -d 'The output directory to store the generated reports in.'
complete -c ort -n "__fish_seen_subcommand_from report" -l report-formats -s f -r -d 'A comma-separated list of report formats to generate, any of [AOSD2.0, AOSD2.1, CtrlXAutomation, CycloneDX, DocBookTemplate, EvaluatedModel, FossID, FossIdSnippet, HtmlTemplate, ManPageTemplate, Opossum, PdfTemplate, PlainTextTemplate, SpdxDocument, StaticHTML, TrustSource, WebApp].'
complete -c ort -n "__fish_seen_subcommand_from report" -l report-formats -s f -r -d 'A comma-separated list of report formats to generate, any of [AOSD2.0, AOSD2.1, CtrlXAutomation, CycloneDX, DocBookTemplate, EvaluatedModel, FossID, FossIdSnippet, HtmlTemplate, ManPageTemplate, Opossum, PdfTemplate, PlainTextTemplate, ScanossSnippet, SpdxDocument, StaticHTML, TrustSource, WebApp].'
complete -c ort -n "__fish_seen_subcommand_from report" -l copyright-garbage-file -r -F -d 'A file containing copyright statements which are marked as garbage. This can make the output inconsistent with the evaluator output but is useful when testing copyright garbage.'
complete -c ort -n "__fish_seen_subcommand_from report" -l custom-license-texts-dir -r -F -d 'A directory which maps custom license IDs to license texts. It should contain one text file per license with the license ID as the filename. A custom license text is used only if its ID has a \'LicenseRef-\' prefix and if the respective license text is not known by ORT.'
complete -c ort -n "__fish_seen_subcommand_from report" -l how-to-fix-text-provider-script -r -F -d 'The path to a Kotlin script which returns an instance of a \'HowToFixTextProvider\'. That provider injects how-to-fix texts in Markdown format for ORT issues.'
Expand Down
39 changes: 39 additions & 0 deletions plugins/reporters/scanoss/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*
* Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
* License-Filename: LICENSE
*/

plugins {
// Apply precompiled plugins.
id("ort-plugin-conventions")
}

dependencies {
api(projects.reporter)

ksp(projects.reporter)

implementation(projects.model)
implementation(projects.plugins.reporters.asciidocReporter)
implementation(projects.plugins.reporters.freemarkerReporter)
implementation(projects.utils.commonUtils)
implementation(projects.utils.ortUtils)

implementation(libs.kotlinx.coroutines)

testImplementation(libs.mockk)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/*
* Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.reporters.scanoss

import java.io.File

import org.ossreviewtoolkit.plugins.api.OrtPlugin
import org.ossreviewtoolkit.plugins.api.PluginDescriptor
import org.ossreviewtoolkit.plugins.reporters.asciidoc.AsciiDocTemplateReporterConfig
import org.ossreviewtoolkit.plugins.reporters.asciidoc.HtmlTemplateReporter
import org.ossreviewtoolkit.reporter.Reporter
import org.ossreviewtoolkit.reporter.ReporterFactory
import org.ossreviewtoolkit.reporter.ReporterInput

@OrtPlugin(
displayName = "SCANOSS Snippet Reporter",
description = "Generates a detailed report of the SCANOSS snippet findings.",
factory = ReporterFactory::class
)
class ScanossSnippetReporter(override val descriptor: PluginDescriptor = ScanossSnippetReporterFactory.descriptor) :
Reporter by delegateReporter {
companion object {
private val delegateReporter = HtmlTemplateReporter(
ScanossSnippetReporterFactory.descriptor,
AsciiDocTemplateReporterConfig(templateIds = listOf("scanoss_snippet"), templatePaths = null)
)
}

override fun generateReport(input: ReporterInput, outputDir: File): List<Result<File>> {
val hasScanossResults = input.ortResult.scanner?.scanResults?.any { it.scanner.name == "SCANOSS" } == true
require(hasScanossResults) { "No SCANOSS scan results have been found." }

return delegateReporter.generateReport(input, outputDir)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Use Unix line endings for Freemarker templates for consistency across platforms.
**/*.ftl text eol=lf
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
[#--
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not the expert here, but do we really need a separate snippet report for SCANOSS? Can't we generalize the existing FossIdSnippetReporter so that it works for any snippet scanner?

I'd like to avoid that each new snippet scanner we introduce support for also requires its own report.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The separate scanner reports approach was discussed with Nikola, who preferred keeping them separate due to concerns about diverging functionality.

However, you make a valid point. Perhaps having a generic reporter that can adapt to each scanner's particularities would be better? @nnobelis, what are your thoughts on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sschuberth :

While it is true that the templates are nearly the same, they are still some differences between FossID and ScanOSS.
Therefore I would like to avoid a bunch of if/else statements in them and keep two different Freemarker templates.

This also make the plugins self-contained, no ?
If they were to share the templates, where the latter would be located ?

Additionally, The reporters themselves seems to be identical, so maybe they could be merged but the same problem would arise: where the class should be located in ORT ?

Copy link
Member

@sschuberth sschuberth Apr 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reporters themselves seems to be identical, so maybe they could be merged but the same problem would arise: where the class should be located in ORT ?

Well, quite simply, we could have a generic "snippet scanner reporter" in ORT that neither carries "FossID" nor "SCANOSS" in its name, but only mentions the used / configured scanner as part of the generated report.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. So we can do that for the Reporter class but as said before, I would like to keep two different templates.

But this is again increasing the complexity of this PR so would you agree to keep this "shared-reporter" approach in a follow-up PR ?

Copy link
Member

@sschuberth sschuberth Apr 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to keep two different templates.

I'd agreed to that if it turns out that we really would need too many conditionals in the template. But I'm not yet convinced of that.

But this is again increasing the complexity of this PR so would you agree to keep this "shared-reporter" approach in a follow-up PR ?

Yes, see my wish here to split this PR.

Copyright (C) 2025 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

SPDX-License-Identifier: Apache-2.0
License-Filename: LICENSE
--]

:publisher: OSS Review Toolkit
[#assign now = .now]
:revdate: ${now?date?iso_local}

:title-page:
:sectnums:
:toc:

= SCANOSS Snippets
List of all the provenances with their files and snippets.
[#list ortResult.scanner.scanResults as scanResult]

[#assign snippetsLimitIssue = helper.getSnippetsLimitIssue()]

[#if snippetsLimitIssue?has_content]
[WARNING]
====
${snippetsLimitIssue}
====
[/#if]

[#if scanResult.provenance.vcsInfo??]
[#assign url = scanResult.provenance.vcsInfo.url]
[#else]
[#assign url = scanResult.provenance.sourceArtifact.url]
[/#if]
== Provenance '${url}'

[#assign summary = scanResult.summary]

Scan start time : ${summary.startTime} +
End time : ${summary.startTime} +
[#if scanResult.provenance.vcsInfo??]
[#assign gitRepoUrl = url]
[#assign gitRevision = scanResult.provenance.vcsInfo.revision]
Git repo URL: ${gitRepoUrl} +
Git revision: ${gitRevision}

[#if gitRepoUrl?contains("github.com")]
[#assign githubBaseURL = '${gitRepoUrl?remove_ending(".git")}/blob/${gitRevision}']
[/#if]
[/#if]

[#list helper.groupSnippetsByFile(summary.snippetFindings) as filePath, snippetFindings ]

[#if gitRepoUrl?? && gitRepoUrl?contains("github.com")]
[#assign localFileURL = '${githubBaseURL}/${filePath}[${filePath}]']
[#else]
[#assign localFileURL = "${filePath}"]
[/#if]
[#assign licenses = helper.collectLicenses(snippetFindings)]

*${localFileURL}* +
License(s):
[#list licenses as license]
${license}[#sep],
[/#list]

[#list helper.groupSnippetsBySourceLines(snippetFindings) as sourceLocation, snippetFinding]
[#assign snippetCount = snippetFinding.snippets?size]

[width=100%]
[cols="1,3,4,1,2"]
|===
| Source Location | pURL | License | Score | Release Date

.${snippetCount*2}+|
Partial match +
${sourceLocation.startLine?c}-${sourceLocation.endLine?c}


[#list snippetFinding.snippets as snippet ]

| ${snippet.purl!""}
| ${snippet.license!""}
| ${snippet.score!""}
| ${snippet.additionalData['release_date']}

4+a|
.Create a snippet choice for this snippet or mark it as false positive
[%collapsible]
====
Add the following lines to the *.ort.yml* file.

To **choose** this snippet:
[source,yaml]
--
snippet_choices:
- provenance:
url: "${scanResult.provenance.vcsInfo.url}"
choices:
- given:
source_location:
path: "${filePath}"
start_line: ${snippetFinding.sourceLocation.startLine?c}
end_line: ${snippetFinding.sourceLocation.endLine?c}
choice:
purl: "${snippet.purl!""}"
reason: "ORIGINAL_FINDING"
comment: "Explain why this snippet choice was made"
--
Or to mark this location has having ONLY **false positives snippets**:
[source,yaml]
--
snippet_choices:
- provenance:
url: "${scanResult.provenance.vcsInfo.url}"
choices:
- given:
source_location:
path: "${filePath}"
start_line: ${snippetFinding.sourceLocation.startLine?c}
end_line: ${snippetFinding.sourceLocation.endLine?c}
choice:
reason: "NO_RELEVANT_FINDING"
comment: "Explain why this location has only false positives snippets"
--
====
[/#list]
|===
[/#list]
[/#list]
[/#list]
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[#ftl]
[#-- @implicitly included --]

[#-- @ftlvariable name="projects" type="kotlin.collections.Set<org.ossreviewtoolkit.reporter.utils.FreemarkerTemplateProcessor.PackageModel>" --]
[#-- @ftlvariable name="packages" type="kotlin.collections.Set<org.ossreviewtoolkit.reporter.utils.FreemarkerTemplateProcessor.PackageModel>" --]
[#-- @ftlvariable name="ortResult" type="org.ossreviewtoolkit.model.OrtResult" --]
[#-- @ftlvariable name="licenseTextProvider" type="org.ossreviewtoolkit.reporter.LicenseTextProvider" --]
[#-- @ftlvariable name="LicenseView" type="org.ossreviewtoolkit.model.licenses.LicenseView" --]
[#-- @ftlvariable name="helper" type="org.ossreviewtoolkit.plugins.reporters.freemarker.FreemarkerTemplateProcessor.TemplateHelper" --]
[#-- @ftlvariable name="projectsAsPackages" type="kotlin.collections.Set<org.ossreviewtoolkit.model.Identifier>" --]
Loading