Skip to content

[Agent] feat: YAML core manifest + CoreManager CLI for libretro buildbot tooling#3247

Merged
JoeMatt merged 12 commits intodevelopfrom
agent/issue-3241-core-manifest
Mar 17, 2026
Merged

[Agent] feat: YAML core manifest + CoreManager CLI for libretro buildbot tooling#3247
JoeMatt merged 12 commits intodevelopfrom
agent/issue-3241-core-manifest

Conversation

@github-actions
Copy link
Contributor

Summary

  • YAML Core Manifest (CoresRetro/RetroArch/scripts/cores.yml) — replaces 4 manually-maintained URL list files with a single source of truth. Each core entry declares its name, ios/tvos availability, appstore eligibility, enabled state, and optional custom filename.
  • Python Generator (Scripts/generate_core_lists.py) — stdlib-only Python 3.8+ script that reads cores.yml and generates all 8 output files (4 url lists + 4 xcfilelists). Subcommands: generate, validate, diff, bootstrap.
  • CoreManager Swift CLI (Scripts/CoreManager/) — Swift Package Manager executable with the same subcommands, using ArgumentParser and a Codable CoreManifest model. Includes XCTest coverage in CoreManifestTests.
  • Documentation (Scripts/CORE_MANAGEMENT.md) — covers problem statement, YAML schema, URL derivation rules, platform differences, and migration path.

Test plan

  • Run python3 Scripts/generate_core_lists.py diff to verify no meaningful diffs vs existing files
  • Run python3 Scripts/generate_core_lists.py generate --dry-run to verify all 8 files would be written
  • Verify dolphin is commented in appstore URL files but present in sideload files
  • Verify vitaquake2 variants are absent from iOS URL files and present in tvOS URL files
  • Verify vice_xscpu64_ibretro_{ios,tvos}.dylib filenames are preserved correctly
  • Optionally: cd Scripts/CoreManager && swift build on macOS

🤖 Generated with Claude Code

@github-actions github-actions bot requested a review from JoeMatt as a code owner March 17, 2026 01:44
@github-actions github-actions bot added the agent-work PR or issue being worked on by the AI agent label Mar 17, 2026
@github-actions github-actions bot enabled auto-merge (squash) March 17, 2026 01:44
@JoeMatt JoeMatt requested a review from Copilot March 17, 2026 01:48
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a single YAML manifest (cores.yml) as the source of truth for libretro core metadata, plus generator tooling (Python + Swift CLI) to reliably regenerate the URL lists and .xcfilelist files used by the RetroArch build tooling.

Changes:

  • Add CoresRetro/RetroArch/scripts/cores.yml to replace multiple manually-maintained core list files.
  • Add Scripts/generate_core_lists.py (stdlib-only) to generate/validate/diff/bootstrap the 8 derived output files.
  • Add Scripts/CoreManager/ SwiftPM CLI (ArgumentParser) with XCTest coverage and accompanying documentation.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
Scripts/generate_core_lists.py Python generator/validator/diff/bootstrap implementation for producing URL lists + xcfilelists from cores.yml.
CoresRetro/RetroArch/scripts/cores.yml New manifest defining core flags (ios/tvos/appstore/enabled) and filename overrides.
Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift Swift Codable model + minimal YAML parsing to load cores.yml.
Scripts/CoreManager/Sources/CoreManager/main.swift Swift CLI subcommands (generate/validate/diff/bootstrap) for core list management.
Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift XCTest coverage for manifest parsing and URL/filename derivation.
Scripts/CoreManager/Package.swift SwiftPM package definition for the CoreManager executable + tests.
Scripts/CORE_MANAGEMENT.md Documentation for schema, tooling, and migration approach.
.changelog/3241.md Changelog fragment describing the added manifest + tooling.

Comment on lines +281 to +285
group.notify(queue: .main) {
semaphore.signal()
}
semaphore.wait()

Comment on lines +352 to +360
let oldSet = Set(oldLines)
let newSet = Set(newLines)

for line in oldLines where !newSet.contains(line) {
print("- \(line)")
}
for line in newLines where !oldSet.contains(line) {
print("+ \(line)")
}
Comment on lines +436 to +442
let iosFile = "\(name)_libretro_ios.dylib"
let tvosFile = "\(name)_libretro_tvos.dylib"
let iosDisabled = !iosEnabled.contains(iosFile)
let tvosDisabled = !tvosEnabled.contains(tvosFile)
let bothDisabled = (entry["ios"] as? Bool == true) && (entry["tvos"] as? Bool == true)
? iosDisabled && tvosDisabled
: (entry["ios"] as? Bool == true ? iosDisabled : tvosDisabled)
Comment on lines +466 to +477
private func coreNameFrom(filename: String) -> (String, String?) {
var base = filename
if base.hasSuffix(".dylib") { base = String(base.dropLast(6)) }
if base.hasSuffix("_libretro_ios") {
return (String(base.dropLast(13)), nil)
} else if base.hasSuffix("_libretro_tvos") {
return (String(base.dropLast(14)), nil)
}
// Platform-neutral
let name = base.replacingOccurrences(of: "_libretro", with: "")
return (name, filename)
}
Comment on lines +180 to +184
} else if let colonRange = content.range(of: ": ") {
// Top-level "key: value"
let key = String(content[content.startIndex..<colonRange.lowerBound])
let value = String(content[colonRange.upperBound...])
result[key.trimmingCharacters(in: .whitespaces)] = parseScalar(value)
Comment on lines +580 to +591
# Check for platform suffix
if base.endswith("_libretro_ios"):
name = base[:-13]
return name, None
elif base.endswith("_libretro_tvos"):
name = base[:-14]
return name, None
else:
# Platform-neutral — return original filename
return base.replace("_libretro", ""), filename


Comment on lines +178 to +179
the Python script. Both tools read the same `cores.yml` and produce identical output.

@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from c079841 to 5e3d3ab Compare March 17, 2026 01:57
@JoeMatt
Copy link
Member

JoeMatt commented Mar 17, 2026

@claude please validate and fix and legit copilot review isses then set copilot to re-review

@github-actions
Copy link
Contributor Author

github-actions bot commented Mar 17, 2026

✅ Fixes Applied

Completed

  • 🚀 Agent started
  • 📖 Read 30 Copilot review comments
  • 🔨 Fixed remaining issues (regenerated files, throttle, bootstrap buildbot)
  • 📤 Pushed to branch

Updated: 2026-03-17 UTC

@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from 5e3d3ab to 98b5ef5 Compare March 17, 2026 02:11
@github-actions github-actions bot added the ai-reviewing AI code review in progress label Mar 17, 2026
@github-actions
Copy link
Contributor Author

✅ Fixes pushed — addressed all 7 Copilot review issues + 3 pre-existing compile errors

Issues fixed:

  • CRITICAL: ValidateCommand deadlock (group.notify+semaphore on main thread → group.wait())
  • MAJOR: printSimpleDiff set-based comparison → CollectionDifference (order-preserving, handles duplicates)
  • MAJOR: BootstrapCommand enabled-check now uses custom filename for platform-neutral cores
  • MAJOR: coreNameFrom() + _core_name_from_filename() handle _ibretro_ upstream typo
  • MAJOR: YAML parser splitYAMLKeyValue() now accepts key:\tvalue (tab) and key: (bare) forms; fixed parseMapping regression where bare key: would bypass nested-block handling
  • MINOR: CORE_MANAGEMENT.md clarifies headers differ between tools

Pre-existing build errors also fixed:

  • @main + main.swift conflict → removed @main, added CoreManager.main() call
  • Linux URLRequest/URLSession — added #if canImport(FoundationNetworking) import
  • var jsonlet json warning

All 13 unit tests pass.

@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from 42752e5 to 23f67e1 Compare March 17, 2026 02:21
@JoeMatt JoeMatt requested a review from Copilot March 17, 2026 02:24
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a single-source-of-truth manifest for libretro buildbot core lists and adds tooling (Python + Swift CLI) to generate/validate/diff/bootstrap the existing URL lists and Xcode .xcfilelist outputs used by the RetroArch module download build phases.

Changes:

  • Add CoresRetro/RetroArch/scripts/cores.yml as the unified core manifest (per-core flags + optional filename overrides).
  • Add Scripts/generate_core_lists.py (stdlib-only) to generate the 4 URL list files + 4 module xcfilelists from the YAML manifest.
  • Add Scripts/CoreManager/ SwiftPM executable (ArgumentParser) with similar subcommands and XCTest coverage, plus documentation and a changelog fragment.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
Scripts/generate_core_lists.py New Python generator/validator/diff/bootstrap tool for producing URL lists and xcfilelists from cores.yml.
CoresRetro/RetroArch/scripts/cores.yml New YAML manifest describing buildbot paths and per-core inclusion/enablement/appstore metadata.
Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift Swift Codable model + lightweight YAML-to-JSON decoding for cores.yml.
Scripts/CoreManager/Sources/CoreManager/main.swift Swift CLI entry point and subcommands (generate/validate/diff/bootstrap).
Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift XCTest coverage for YAML decoding and URL/filename derivation logic.
Scripts/CoreManager/Package.swift SwiftPM package definition for the CoreManager executable + tests.
Scripts/CORE_MANAGEMENT.md Documentation for the new manifest schema, generation rules, and migration path.
.changelog/3241.md Changelog fragment describing the new manifest and tooling additions.

Comment on lines +387 to +395
# Omit globally-disabled cores from xcfilelists entirely
if not core.enabled:
continue

# Appstore-excluded cores are omitted from appstore xcfilelists
if appstore and not core.appstore:
continue

lines.append(f"{XCFILELIST_PREFIX}{fname}\n")
Comment on lines +580 to +594
# Check for platform suffix (including known upstream typo: _ibretro instead of _libretro)
if base.endswith("_libretro_ios"):
return base[:-13], None
elif base.endswith("_libretro_tvos"):
return base[:-14], None
elif base.endswith("_ibretro_ios"):
# Handle upstream typo: missing 'l' in 'libretro'
return base[:-12], None
elif base.endswith("_ibretro_tvos"):
# Handle upstream typo: missing 'l' in 'libretro'
return base[:-13], None
else:
# Platform-neutral — return original filename
return base.replace("_libretro", ""), filename

Comment on lines +13 to +16
# appstore - allowed in App Store builds (false = excluded from appstore variants)
# enabled - if false, written as a commented line (#url) in all generated files
# filename - ONLY set for platform-neutral filenames (no _ios/_tvos suffix)
# e.g., "flycast_libretro.dylib"
Comment on lines +102 to +106
# Auto-generated by Scripts/CoreManager — do not edit manually
# Edit CoresRetro/RetroArch/scripts/cores.yml and re-run:
# swift run CoreManager generate

"""
Comment on lines +141 to +149
// Disabled cores are omitted entirely from xcfilelists
guard core.enabled else { continue }
// Appstore-excluded cores are omitted from appstore xcfilelists
if appstore && !core.appstore { continue }

let fname = platform == "ios" ? core.iosFilename : core.tvosFilename
lines.append("\(xcfilelistPrefix)\(fname)\n")
}
return lines.joined()
Comment on lines +466 to +483
private func coreNameFrom(filename: String) -> (String, String?) {
var base = filename
if base.hasSuffix(".dylib") { base = String(base.dropLast(6)) }
if base.hasSuffix("_libretro_ios") {
return (String(base.dropLast(13)), nil)
} else if base.hasSuffix("_libretro_tvos") {
return (String(base.dropLast(14)), nil)
} else if base.hasSuffix("_ibretro_ios") {
// Handle upstream typo: missing 'l' in 'libretro'
return (String(base.dropLast(12)), nil)
} else if base.hasSuffix("_ibretro_tvos") {
// Handle upstream typo: missing 'l' in 'libretro'
return (String(base.dropLast(13)), nil)
}
// Platform-neutral
let name = base.replacingOccurrences(of: "_libretro", with: "")
return (name, filename)
}
Comment on lines +132 to +133
// Delegate to the Python script for heavy lifting in production.
// For the Swift tool we use a simple line-by-line parser.
Comment on lines +331 to +333
"# Auto-generated by Scripts/generate_core_lists.py\n"
"# Do not edit manually — edit CoresRetro/RetroArch/scripts/cores.yml instead\n"
"# and re-run: python3 Scripts/generate_core_lists.py generate\n"
@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from 23f67e1 to 2ab9d8f Compare March 17, 2026 02:39
@JoeMatt
Copy link
Member

JoeMatt commented Mar 17, 2026

@claude please validate and fix any and all valid copilot review issues then re-request copilot review

@github-actions
Copy link
Contributor Author

✅ Fixes pushed — addressed all 14 Copilot review issues:

main.swift + generate_core_lists.py:

  • Removed spaced headers from URL files (they feed xargs curl -O — spaces break tokenization)
  • generateXCFilelist / generate_xcfilelist: disabled/excluded cores now emit commented entries (#path) instead of being omitted entirely, matching existing file convention
  • coreNameFrom / _core_name_from_filename: _ibretro_* (upstream typo) now returns the original filename as a custom override so bootstrap can round-trip correctly
  • BootstrapCommand / cmd_bootstrap: tracks ios_filename/tvos_filename separately for per-platform overrides; uses correct filename in enabled check; emits ios_filename/tvos_filename keys in YAML output

CoreManifest.swift:

  • Removed misleading "delegates to Python" comment

cores.yml:

  • Clarified enabled comment to explicitly mention xcfilelists

CORE_MANAGEMENT.md:

  • Updated enabled table row (no longer says "omitted from xcfilelists")
  • Updated xcfilelist output descriptions to say "commented" not "enabled only"

Note: Three Copilot issues were already addressed in the existing code (ValidateCommand uses group.wait() not semaphore, DiffCommand uses CollectionDifference not set-diff, CoreManifest YAML parser already uses firstIndex(of:) not range(of:)).

@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from 8c6fa04 to 711914e Compare March 17, 2026 03:06
@JoeMatt JoeMatt requested a review from Copilot March 17, 2026 03:07
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a single-source-of-truth cores.yml manifest for libretro buildbot cores, plus tooling (Python + Swift CLI) to generate/validate/diff the existing URL lists and Xcode .xcfilelist outputs used by the RetroArch core download/build pipeline.

Changes:

  • Add CoresRetro/RetroArch/scripts/cores.yml manifest describing cores, platform availability, appstore eligibility, and filename overrides.
  • Add Scripts/generate_core_lists.py (stdlib-only) to generate + validate + diff + bootstrap the existing list files from the manifest.
  • Add Scripts/CoreManager/ SwiftPM CLI (ArgumentParser) with matching subcommands and XCTest coverage, plus documentation and changelog fragment.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
Scripts/generate_core_lists.py New Python generator/validator/differ/bootstrapper for URL lists + xcfilelists driven by cores.yml.
Scripts/CoreManager/Sources/CoreManager/main.swift Swift CLI entry point + subcommands to generate/validate/diff/bootstrap manifest-driven outputs.
Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift Swift Codable model + lightweight YAML parsing to load cores.yml.
Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift XCTest coverage for manifest parsing, filename derivation, URL generation, and filtering semantics.
Scripts/CoreManager/Package.swift SwiftPM package definition for CoreManager executable + tests.
Scripts/CORE_MANAGEMENT.md Documentation describing the manifest schema, generator behavior, and migration path.
CoresRetro/RetroArch/scripts/cores.yml The new unified YAML manifest of libretro cores and buildbot configuration.
.changelog/3241.md Changelog fragment announcing the new manifest and tooling.

Comment on lines +440 to +447
def cmd_validate(manifest: CoreManifest, verbose: bool) -> None:
"""HTTP HEAD check all enabled URLs in the manifest."""
errors = 0
checked = 0

all_ios = manifest.ios_cores(appstore=False)
all_tvos = manifest.tvos_cores(appstore=False)

Comment on lines +497 to +501
def cmd_diff(manifest: CoreManifest, verbose: bool) -> None:
"""Show diff between what would be generated and current files on disk."""
all_ios = manifest.ios_cores(appstore=False)
all_tvos = manifest.tvos_cores(appstore=False)

for (key, content) in outputs {
guard let url = files[key] else { continue }
if dryRun {
let lineCount = content.components(separatedBy: .newlines).count
Comment on lines +226 to +230
@OptionGroup var manifestOptions: ManifestOptions

@Flag(name: .shortAndLong, help: "Show additional output")
var verbose: Bool = false

Comment on lines +404 to +407
func register(filename: String, platform: String, enabled: Bool) {
let (name, custom) = coreNameFrom(filename: filename)
if all[name] == nil {
all[name] = [
Comment on lines +41 to +42
/// When false, the core is commented out in appstore url lists and
/// omitted from appstore xcfilelists.
Comment on lines +46 to +47
/// When false, the core appears as a commented line in ALL generated files
/// and is omitted from all xcfilelists.
Comment on lines +339 to +345
def generate_url_file(
cores: List[CoreEntry],
buildbot: BuildbotConfig,
platform: str, # 'ios' | 'tvos'
appstore: bool,
verbose: bool = False,
) -> str:
@JoeMatt
Copy link
Member

JoeMatt commented Mar 17, 2026

@claude fix the copilot review issues please

@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch 2 times, most recently from e109bbe to 51cced5 Compare March 17, 2026 03:24
@github-actions
Copy link
Contributor Author

✅ Fixes pushed — addressed all actionable Copilot review issues:

  • main.swift: dry-run line count now uses split(whereSeparator:) to avoid off-by-1 from trailing newline
  • main.swift: ValidateCommand.verbose flag implemented — shows core name in output when enabled
  • main.swift: removed unused enabled: parameter from nested register() function
  • CoreManifest.swift: updated appstore and enabled field docstrings to say "commented out" (not "omitted") matching actual generator behavior
  • generate_core_lists.py: removed unused verbose param from generate_url_file()
  • generate_core_lists.py: cmd_validate verbose mode now prints [SKIP] for disabled cores
  • generate_core_lists.py: cmd_diff verbose mode now prints [OK] confirmation per file

Several other Copilot comments were already resolved in the current code (deadlock fix, _ibretro_* filename handling, xcfilelist commented-line behavior, YAML parser robustness).

@JoeMatt JoeMatt requested a review from Copilot March 17, 2026 03:28
@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch 14 times, most recently from 1e94e40 to d138b9b Compare March 17, 2026 08:16
github-actions bot and others added 12 commits March 17, 2026 08:18
Replace 4 manually-maintained urls.txt / xcfilelist files with a
single cores.yml source of truth. Add Python generator script and
Swift CoreManager CLI package for regenerating all output files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CRITICAL: fix ValidateCommand deadlock (group.notify+semaphore on main
  thread → group.wait() instead)
- MAJOR: replace set-based printSimpleDiff with CollectionDifference for
  ordering/duplicate correctness
- MAJOR: BootstrapCommand enabled-check now uses custom filename for
  platform-neutral cores instead of hardcoded _libretro_{ios,tvos}
- MAJOR: coreNameFrom() + _core_name_from_filename() handle _ibretro_
  upstream typo (missing 'l' in 'libretro')
- MAJOR: YAML parser splitYAMLKeyValue() accepts key:\tvalue and key:
  (not just key: value); fix parseMapping regression from splitYAMLKeyValue
  matching bare key: before nested-block branch
- MINOR: clarify CORE_MANAGEMENT.md that headers differ between tools
- Fix pre-existing @main/main.swift conflict; add FoundationNetworking
  import for Linux; var→let for unmutated json binding
- All 13 unit tests pass

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove header from URL files (spaces break `xargs curl -O`);
  headers are kept only for xcfilelists
- xcfilelists now emit commented entries for disabled/excluded cores
  instead of omitting them, matching existing file convention
- coreNameFrom/_core_name_from_filename: return filename for _ibretro_*
  typo patterns so bootstrap can round-trip them correctly
- Bootstrap: track ios_filename/tvos_filename overrides separately and
  emit them in YAML output; use per-platform names in enabled check
- Remove misleading "delegates to Python" comment in CoreManifest.swift
- Update cores.yml comment and CORE_MANAGEMENT.md to reflect that
  disabled cores are commented (not omitted) in xcfilelists

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…string accuracy, dry-run line count

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_path, docstrings

- fix(diff --verbose): was always printing [OK] even when diffs found; now
  tracks per-file diff status and prints [OK  ] or [DIFF] accurately
- fix(bootstrap): cmd_bootstrap ignored the --manifest argument and always
  used the hardcoded SCRIPTS_DIR; now derives scripts dir from manifest_path
- docs(ios_cores/tvos_cores): clarify the appstore parameter semantics in
  both Python and Swift — it filters to eligible-only when True, but callers
  should pass False (default) so generators can emit excluded cores as
  commented lines rather than dropping them
- docs(CoreEntry): tighten appstore/enabled docstrings to say 'commented-out
  line' rather than 'omitted', matching the actual generator behaviour

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…otstrap buildbot

- Run generator to sync url/xcfilelist files with cores.yml manifest:
  removes duplicate vitaquake2/mcsoftserve entries in existing files,
  removes iOS vitaquake2 entries (ios: false in manifest),
  alphabetical ordering consistent with generator output
- Add URL request throttling in Swift ValidateCommand (semaphore=8,
  httpMaximumConnectionsPerHost=4) to prevent buildbot overload
- Add buildbot: section to Python cmd_bootstrap output so bootstrap
  produces a complete, usable cores.yml starter (matches Swift behavior)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@github-actions github-actions bot force-pushed the agent/issue-3241-core-manifest branch from d138b9b to 8c6d61d Compare March 17, 2026 08:18
@JoeMatt JoeMatt merged commit 3dba467 into develop Mar 17, 2026
11 checks passed
@JoeMatt JoeMatt deleted the agent/issue-3241-core-manifest branch March 17, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-work PR or issue being worked on by the AI agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore improved libretro buildbot core management tooling (replace urls.txt + xcfilelist)

2 participants