AI-generated summary of scan deltas in natural language

[wkl3nk]

I think this is not so hard to implement and could introduce AI into ORT Server.

Motivation

Imagine your repository scans have consistently shown a status over the past few weeks — even though your team has been pushing new revisions every couple of days. Then, suddenly, the scan status turns to or overnight.

Currently, identifying what changed between two revisions of the repository is a manual and time-consuming process. An expert needs to open both reports, compare them side by side, and try to pinpoint what triggered the red status.

Wouldn’t it be helpful if the user interface provided an AI-generated summary in natural language highlighting exactly what changed between the last successful scan and the current one — and likely explaining why the status turned red?

Example AI-generated summary:

Compliance Scan Delta (v1.8.1 → v1.8.2)

✅ Resolved 2 GPL-3.0 license violations (in some-lib and legacy-util)
❗ Introduced 1 new policy violation: AGPL-3.0 in experimental-ui-lib
❗ 1 new vulnerability detected (CVE-2023-4567 in json-parser, severity: HIGH)
✅ Removed dependency to abc:1.2.3

⚠️ Legal and Security teams should review the AGPL and CVE before release approval.

What result differences could be part of the AI-generated summary

• Differences in individual stages of the overall scan:
  "In this revision, the Analyzer failed."
• Differences in dependencies:
  "New dependencies to abc:1.2.3 and def:4.5.6 were introduced."
• Differences in known vulnerabilities:
  "1 new vulnerability detected (CVE-2023-4567 in json-parser, severity: HIGH)"
• Differences in policy violations:
  "Introduced 1 new policy violation: AGPL-3.0 in experimental-ui-lib."
• Differences in issues:
  "A new issue popped up: Provenance for abc:1.2.3 could not be found."
• Differences of execution time:
  "The execution time for the Analyzer stage doubled from 1 minute to 2 minutes."

... more to be added.
This can be developed step by step, adding more and more result differences over time.

How to implement this?

• Well, we already have all the data, in ort-result.yml and in ORT Server database tables.
• It should be easy to compare the data with the previous run in order to generate a diff in some kind of text format.
• Then this text would be slightly augmented to act as prompt for a LLM like ChatGPT 4o (pluggable) like "Create a summary in natural language of the differences between ... these are as follows .... (include list of diffs)".

[sschuberth]

Just pointing out https://github.com/JetBrains/koog/ as a possible framework here.