Skip to content

Explore improved libretro buildbot core management tooling (replace urls.txt + xcfilelist) #3241

@github-actions

Description

@github-actions

Overview

The current libretro buildbot → .framework management system uses manually-maintained .txt URL lists and .xcfilelist files (~130 entries × 4 build variants). This is difficult to maintain, error-prone, and has no validation tooling.

This issue is a research + prototyping task to explore a better system. No breaking changes should land in this PR — the goal is a proposal and/or a small working proof-of-concept.

Part of #3234


Current Pain Points (from @JoeMatt)

  1. urls.txt, urls-appstore.txt, urls-tv.txt, urls-appstore-tv.txt — four files that must be kept in sync, no diffing between them, no validation that URLs are reachable
  2. output_*.xcfilelist files must be manually updated when cores are added/removed
  3. No tooling to detect: (a) dead/404 URLs, (b) cores present in urls.txt but missing from Core.plist, (c) cores in Core.plist but not fetched
  4. No easy way to flag a core as "do not include in App Store build" at the URL-fetch layer (only enforced at the Xcode build phase via xcfilelist, but the dylib still gets fetched)
  5. Version pinning is implicit (uses latest/ path) — no easy way to pin or audit core versions

Acceptance Criteria

This is an exploratory PR. Deliver:

1. A written proposal (Scripts/CORE_MANAGEMENT.md)

Document the proposed new system. Options to evaluate:

Option A — YAML/TOML manifest
Replace 4 txt files with a single cores.yaml manifest:

cores:
  - name: fceumm
    ios: true
    tvos: true
    appstore: true
    license: GPL-2.0-only
    upstream: https://github.com/libretro/libretro-fceumm
  - name: flycast
    ios: true
    tvos: false
    appstore: false
    license: GPL-2.0-only
    appstore_excluded_reason: "GPL-2.0 dylib distribution not permitted in App Store without source offer"

A code generator script then produces the 4 .txt files and xcfilelists.

Option B — Swift CLI tool
A lightweight Swift executable (CoreManager) in Scripts/CoreManager/ that:

  • Validates URLs against the buildbot
  • Diffs current vs. updated core list
  • Generates txt/xcfilelist files
  • Reports license compliance gaps

Option C — Enhanced shell scripts + JSON
Minimal change: replace .txt files with a JSON source of truth, add a validation shell script.

Option D — Fastlane lane
A Fastfile lane validate_cores that checks URLs, warns on license gaps, regenerates xcfilelists.

2. Proof-of-concept implementation

Implement whichever option is chosen (or a hybrid) as a working prototype:

  • Replaces at least the iOS non-appstore variant as a test
  • Can validate that all URLs in the manifest are reachable (HTTP HEAD check)
  • Can detect cores in urls.txt with no Core.plist entry
  • Has a --dry-run mode

3. Migration path

  • Document how to migrate existing 4-file setup to the new system without disrupting builds
  • Ensure existing Xcode build phases still work (xcfilelist files can be generated by the new tool)

Out of Scope for This PR

  • Migrating all 130 cores to the new system (follow-up task)
  • Changing the actual core selection / set of cores shipped
  • Modifying project.pbxproj

Files to Create

  • Scripts/CORE_MANAGEMENT.md — proposal doc
  • Scripts/CoreManager/ (or Scripts/cores.yaml + generator script) — prototype

Branch

agent/issue-3234-buildbot-tooling or similar. Target develop.

Metadata

Metadata

Assignees

No one assigned

    Labels

    appstoreRelated to Apple App Store buildsciRelated to CI/CD pipelinescore-relatedemulation specificdevimplementation, code, developmentenhancementimprovements, enhancements, new features, additions

    Type

    No type

    Projects

    Status

    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions