[Agent] feat: YAML core manifest + CoreManager CLI for libretro buildbot tooling#3247
[Agent] feat: YAML core manifest + CoreManager CLI for libretro buildbot tooling#3247
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a single YAML manifest (cores.yml) as the source of truth for libretro core metadata, plus generator tooling (Python + Swift CLI) to reliably regenerate the URL lists and .xcfilelist files used by the RetroArch build tooling.
Changes:
- Add
CoresRetro/RetroArch/scripts/cores.ymlto replace multiple manually-maintained core list files. - Add
Scripts/generate_core_lists.py(stdlib-only) to generate/validate/diff/bootstrap the 8 derived output files. - Add
Scripts/CoreManager/SwiftPM CLI (ArgumentParser) with XCTest coverage and accompanying documentation.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Scripts/generate_core_lists.py | Python generator/validator/diff/bootstrap implementation for producing URL lists + xcfilelists from cores.yml. |
| CoresRetro/RetroArch/scripts/cores.yml | New manifest defining core flags (ios/tvos/appstore/enabled) and filename overrides. |
| Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift | Swift Codable model + minimal YAML parsing to load cores.yml. |
| Scripts/CoreManager/Sources/CoreManager/main.swift | Swift CLI subcommands (generate/validate/diff/bootstrap) for core list management. |
| Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift | XCTest coverage for manifest parsing and URL/filename derivation. |
| Scripts/CoreManager/Package.swift | SwiftPM package definition for the CoreManager executable + tests. |
| Scripts/CORE_MANAGEMENT.md | Documentation for schema, tooling, and migration approach. |
| .changelog/3241.md | Changelog fragment describing the added manifest + tooling. |
| group.notify(queue: .main) { | ||
| semaphore.signal() | ||
| } | ||
| semaphore.wait() | ||
|
|
| let oldSet = Set(oldLines) | ||
| let newSet = Set(newLines) | ||
|
|
||
| for line in oldLines where !newSet.contains(line) { | ||
| print("- \(line)") | ||
| } | ||
| for line in newLines where !oldSet.contains(line) { | ||
| print("+ \(line)") | ||
| } |
| let iosFile = "\(name)_libretro_ios.dylib" | ||
| let tvosFile = "\(name)_libretro_tvos.dylib" | ||
| let iosDisabled = !iosEnabled.contains(iosFile) | ||
| let tvosDisabled = !tvosEnabled.contains(tvosFile) | ||
| let bothDisabled = (entry["ios"] as? Bool == true) && (entry["tvos"] as? Bool == true) | ||
| ? iosDisabled && tvosDisabled | ||
| : (entry["ios"] as? Bool == true ? iosDisabled : tvosDisabled) |
| private func coreNameFrom(filename: String) -> (String, String?) { | ||
| var base = filename | ||
| if base.hasSuffix(".dylib") { base = String(base.dropLast(6)) } | ||
| if base.hasSuffix("_libretro_ios") { | ||
| return (String(base.dropLast(13)), nil) | ||
| } else if base.hasSuffix("_libretro_tvos") { | ||
| return (String(base.dropLast(14)), nil) | ||
| } | ||
| // Platform-neutral | ||
| let name = base.replacingOccurrences(of: "_libretro", with: "") | ||
| return (name, filename) | ||
| } |
| } else if let colonRange = content.range(of: ": ") { | ||
| // Top-level "key: value" | ||
| let key = String(content[content.startIndex..<colonRange.lowerBound]) | ||
| let value = String(content[colonRange.upperBound...]) | ||
| result[key.trimmingCharacters(in: .whitespaces)] = parseScalar(value) |
Scripts/generate_core_lists.py
Outdated
| # Check for platform suffix | ||
| if base.endswith("_libretro_ios"): | ||
| name = base[:-13] | ||
| return name, None | ||
| elif base.endswith("_libretro_tvos"): | ||
| name = base[:-14] | ||
| return name, None | ||
| else: | ||
| # Platform-neutral — return original filename | ||
| return base.replace("_libretro", ""), filename | ||
|
|
||
|
|
Scripts/CORE_MANAGEMENT.md
Outdated
| the Python script. Both tools read the same `cores.yml` and produce identical output. | ||
|
|
c079841 to
5e3d3ab
Compare
|
@claude please validate and fix and legit copilot review isses then set copilot to re-review |
✅ Fixes AppliedCompleted
Updated: 2026-03-17 UTC |
5e3d3ab to
98b5ef5
Compare
|
✅ Fixes pushed — addressed all 7 Copilot review issues + 3 pre-existing compile errors Issues fixed:
Pre-existing build errors also fixed:
All 13 unit tests pass. |
42752e5 to
23f67e1
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a single-source-of-truth manifest for libretro buildbot core lists and adds tooling (Python + Swift CLI) to generate/validate/diff/bootstrap the existing URL lists and Xcode .xcfilelist outputs used by the RetroArch module download build phases.
Changes:
- Add
CoresRetro/RetroArch/scripts/cores.ymlas the unified core manifest (per-core flags + optional filename overrides). - Add
Scripts/generate_core_lists.py(stdlib-only) to generate the 4 URL list files + 4 module xcfilelists from the YAML manifest. - Add
Scripts/CoreManager/SwiftPM executable (ArgumentParser) with similar subcommands and XCTest coverage, plus documentation and a changelog fragment.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| Scripts/generate_core_lists.py | New Python generator/validator/diff/bootstrap tool for producing URL lists and xcfilelists from cores.yml. |
| CoresRetro/RetroArch/scripts/cores.yml | New YAML manifest describing buildbot paths and per-core inclusion/enablement/appstore metadata. |
| Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift | Swift Codable model + lightweight YAML-to-JSON decoding for cores.yml. |
| Scripts/CoreManager/Sources/CoreManager/main.swift | Swift CLI entry point and subcommands (generate/validate/diff/bootstrap). |
| Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift | XCTest coverage for YAML decoding and URL/filename derivation logic. |
| Scripts/CoreManager/Package.swift | SwiftPM package definition for the CoreManager executable + tests. |
| Scripts/CORE_MANAGEMENT.md | Documentation for the new manifest schema, generation rules, and migration path. |
| .changelog/3241.md | Changelog fragment describing the new manifest and tooling additions. |
Scripts/generate_core_lists.py
Outdated
| # Omit globally-disabled cores from xcfilelists entirely | ||
| if not core.enabled: | ||
| continue | ||
|
|
||
| # Appstore-excluded cores are omitted from appstore xcfilelists | ||
| if appstore and not core.appstore: | ||
| continue | ||
|
|
||
| lines.append(f"{XCFILELIST_PREFIX}{fname}\n") |
| # Check for platform suffix (including known upstream typo: _ibretro instead of _libretro) | ||
| if base.endswith("_libretro_ios"): | ||
| return base[:-13], None | ||
| elif base.endswith("_libretro_tvos"): | ||
| return base[:-14], None | ||
| elif base.endswith("_ibretro_ios"): | ||
| # Handle upstream typo: missing 'l' in 'libretro' | ||
| return base[:-12], None | ||
| elif base.endswith("_ibretro_tvos"): | ||
| # Handle upstream typo: missing 'l' in 'libretro' | ||
| return base[:-13], None | ||
| else: | ||
| # Platform-neutral — return original filename | ||
| return base.replace("_libretro", ""), filename | ||
|
|
| # appstore - allowed in App Store builds (false = excluded from appstore variants) | ||
| # enabled - if false, written as a commented line (#url) in all generated files | ||
| # filename - ONLY set for platform-neutral filenames (no _ios/_tvos suffix) | ||
| # e.g., "flycast_libretro.dylib" |
| # Auto-generated by Scripts/CoreManager — do not edit manually | ||
| # Edit CoresRetro/RetroArch/scripts/cores.yml and re-run: | ||
| # swift run CoreManager generate | ||
|
|
||
| """ |
| // Disabled cores are omitted entirely from xcfilelists | ||
| guard core.enabled else { continue } | ||
| // Appstore-excluded cores are omitted from appstore xcfilelists | ||
| if appstore && !core.appstore { continue } | ||
|
|
||
| let fname = platform == "ios" ? core.iosFilename : core.tvosFilename | ||
| lines.append("\(xcfilelistPrefix)\(fname)\n") | ||
| } | ||
| return lines.joined() |
| private func coreNameFrom(filename: String) -> (String, String?) { | ||
| var base = filename | ||
| if base.hasSuffix(".dylib") { base = String(base.dropLast(6)) } | ||
| if base.hasSuffix("_libretro_ios") { | ||
| return (String(base.dropLast(13)), nil) | ||
| } else if base.hasSuffix("_libretro_tvos") { | ||
| return (String(base.dropLast(14)), nil) | ||
| } else if base.hasSuffix("_ibretro_ios") { | ||
| // Handle upstream typo: missing 'l' in 'libretro' | ||
| return (String(base.dropLast(12)), nil) | ||
| } else if base.hasSuffix("_ibretro_tvos") { | ||
| // Handle upstream typo: missing 'l' in 'libretro' | ||
| return (String(base.dropLast(13)), nil) | ||
| } | ||
| // Platform-neutral | ||
| let name = base.replacingOccurrences(of: "_libretro", with: "") | ||
| return (name, filename) | ||
| } |
| // Delegate to the Python script for heavy lifting in production. | ||
| // For the Swift tool we use a simple line-by-line parser. |
Scripts/generate_core_lists.py
Outdated
| "# Auto-generated by Scripts/generate_core_lists.py\n" | ||
| "# Do not edit manually — edit CoresRetro/RetroArch/scripts/cores.yml instead\n" | ||
| "# and re-run: python3 Scripts/generate_core_lists.py generate\n" |
23f67e1 to
2ab9d8f
Compare
|
@claude please validate and fix any and all valid copilot review issues then re-request copilot review |
|
✅ Fixes pushed — addressed all 14 Copilot review issues: main.swift + generate_core_lists.py:
CoreManifest.swift:
cores.yml:
CORE_MANAGEMENT.md:
Note: Three Copilot issues were already addressed in the existing code (ValidateCommand uses |
8c6fa04 to
711914e
Compare
There was a problem hiding this comment.
Pull request overview
Introduces a single-source-of-truth cores.yml manifest for libretro buildbot cores, plus tooling (Python + Swift CLI) to generate/validate/diff the existing URL lists and Xcode .xcfilelist outputs used by the RetroArch core download/build pipeline.
Changes:
- Add
CoresRetro/RetroArch/scripts/cores.ymlmanifest describing cores, platform availability, appstore eligibility, and filename overrides. - Add
Scripts/generate_core_lists.py(stdlib-only) to generate + validate + diff + bootstrap the existing list files from the manifest. - Add
Scripts/CoreManager/SwiftPM CLI (ArgumentParser) with matching subcommands and XCTest coverage, plus documentation and changelog fragment.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| Scripts/generate_core_lists.py | New Python generator/validator/differ/bootstrapper for URL lists + xcfilelists driven by cores.yml. |
| Scripts/CoreManager/Sources/CoreManager/main.swift | Swift CLI entry point + subcommands to generate/validate/diff/bootstrap manifest-driven outputs. |
| Scripts/CoreManager/Sources/CoreManager/CoreManifest.swift | Swift Codable model + lightweight YAML parsing to load cores.yml. |
| Scripts/CoreManager/Tests/CoreManagerTests/CoreManifestTests.swift | XCTest coverage for manifest parsing, filename derivation, URL generation, and filtering semantics. |
| Scripts/CoreManager/Package.swift | SwiftPM package definition for CoreManager executable + tests. |
| Scripts/CORE_MANAGEMENT.md | Documentation describing the manifest schema, generator behavior, and migration path. |
| CoresRetro/RetroArch/scripts/cores.yml | The new unified YAML manifest of libretro cores and buildbot configuration. |
| .changelog/3241.md | Changelog fragment announcing the new manifest and tooling. |
| def cmd_validate(manifest: CoreManifest, verbose: bool) -> None: | ||
| """HTTP HEAD check all enabled URLs in the manifest.""" | ||
| errors = 0 | ||
| checked = 0 | ||
|
|
||
| all_ios = manifest.ios_cores(appstore=False) | ||
| all_tvos = manifest.tvos_cores(appstore=False) | ||
|
|
| def cmd_diff(manifest: CoreManifest, verbose: bool) -> None: | ||
| """Show diff between what would be generated and current files on disk.""" | ||
| all_ios = manifest.ios_cores(appstore=False) | ||
| all_tvos = manifest.tvos_cores(appstore=False) | ||
|
|
| for (key, content) in outputs { | ||
| guard let url = files[key] else { continue } | ||
| if dryRun { | ||
| let lineCount = content.components(separatedBy: .newlines).count |
| @OptionGroup var manifestOptions: ManifestOptions | ||
|
|
||
| @Flag(name: .shortAndLong, help: "Show additional output") | ||
| var verbose: Bool = false | ||
|
|
| func register(filename: String, platform: String, enabled: Bool) { | ||
| let (name, custom) = coreNameFrom(filename: filename) | ||
| if all[name] == nil { | ||
| all[name] = [ |
| /// When false, the core is commented out in appstore url lists and | ||
| /// omitted from appstore xcfilelists. |
| /// When false, the core appears as a commented line in ALL generated files | ||
| /// and is omitted from all xcfilelists. |
| def generate_url_file( | ||
| cores: List[CoreEntry], | ||
| buildbot: BuildbotConfig, | ||
| platform: str, # 'ios' | 'tvos' | ||
| appstore: bool, | ||
| verbose: bool = False, | ||
| ) -> str: |
|
@claude fix the copilot review issues please |
e109bbe to
51cced5
Compare
|
✅ Fixes pushed — addressed all actionable Copilot review issues:
Several other Copilot comments were already resolved in the current code (deadlock fix, |
1e94e40 to
d138b9b
Compare
Replace 4 manually-maintained urls.txt / xcfilelist files with a single cores.yml source of truth. Add Python generator script and Swift CoreManager CLI package for regenerating all output files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CRITICAL: fix ValidateCommand deadlock (group.notify+semaphore on main
thread → group.wait() instead)
- MAJOR: replace set-based printSimpleDiff with CollectionDifference for
ordering/duplicate correctness
- MAJOR: BootstrapCommand enabled-check now uses custom filename for
platform-neutral cores instead of hardcoded _libretro_{ios,tvos}
- MAJOR: coreNameFrom() + _core_name_from_filename() handle _ibretro_
upstream typo (missing 'l' in 'libretro')
- MAJOR: YAML parser splitYAMLKeyValue() accepts key:\tvalue and key:
(not just key: value); fix parseMapping regression from splitYAMLKeyValue
matching bare key: before nested-block branch
- MINOR: clarify CORE_MANAGEMENT.md that headers differ between tools
- Fix pre-existing @main/main.swift conflict; add FoundationNetworking
import for Linux; var→let for unmutated json binding
- All 13 unit tests pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove header from URL files (spaces break `xargs curl -O`); headers are kept only for xcfilelists - xcfilelists now emit commented entries for disabled/excluded cores instead of omitting them, matching existing file convention - coreNameFrom/_core_name_from_filename: return filename for _ibretro_* typo patterns so bootstrap can round-trip them correctly - Bootstrap: track ios_filename/tvos_filename overrides separately and emit them in YAML output; use per-platform names in enabled check - Remove misleading "delegates to Python" comment in CoreManifest.swift - Update cores.yml comment and CORE_MANAGEMENT.md to reflect that disabled cores are commented (not omitted) in xcfilelists Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…string accuracy, dry-run line count Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_path, docstrings - fix(diff --verbose): was always printing [OK] even when diffs found; now tracks per-file diff status and prints [OK ] or [DIFF] accurately - fix(bootstrap): cmd_bootstrap ignored the --manifest argument and always used the hardcoded SCRIPTS_DIR; now derives scripts dir from manifest_path - docs(ios_cores/tvos_cores): clarify the appstore parameter semantics in both Python and Swift — it filters to eligible-only when True, but callers should pass False (default) so generators can emit excluded cores as commented lines rather than dropping them - docs(CoreEntry): tighten appstore/enabled docstrings to say 'commented-out line' rather than 'omitted', matching the actual generator behaviour Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…otstrap buildbot - Run generator to sync url/xcfilelist files with cores.yml manifest: removes duplicate vitaquake2/mcsoftserve entries in existing files, removes iOS vitaquake2 entries (ios: false in manifest), alphabetical ordering consistent with generator output - Add URL request throttling in Swift ValidateCommand (semaphore=8, httpMaximumConnectionsPerHost=4) to prevent buildbot overload - Add buildbot: section to Python cmd_bootstrap output so bootstrap produces a complete, usable cores.yml starter (matches Swift behavior) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
d138b9b to
8c6d61d
Compare
Summary
CoresRetro/RetroArch/scripts/cores.yml) — replaces 4 manually-maintained URL list files with a single source of truth. Each core entry declares its name, ios/tvos availability, appstore eligibility, enabled state, and optional custom filename.Scripts/generate_core_lists.py) — stdlib-only Python 3.8+ script that readscores.ymland generates all 8 output files (4 url lists + 4 xcfilelists). Subcommands:generate,validate,diff,bootstrap.Scripts/CoreManager/) — Swift Package Manager executable with the same subcommands, using ArgumentParser and a Codable CoreManifest model. Includes XCTest coverage inCoreManifestTests.Scripts/CORE_MANAGEMENT.md) — covers problem statement, YAML schema, URL derivation rules, platform differences, and migration path.Test plan
python3 Scripts/generate_core_lists.py diffto verify no meaningful diffs vs existing filespython3 Scripts/generate_core_lists.py generate --dry-runto verify all 8 files would be writtendolphinis commented in appstore URL files but present in sideload filesvitaquake2variants are absent from iOS URL files and present in tvOS URL filesvice_xscpu64_ibretro_{ios,tvos}.dylibfilenames are preserved correctlycd Scripts/CoreManager && swift buildon macOS🤖 Generated with Claude Code