Roadmap

For smaller items, please see the open issue list.

Symbol Key:

• {❙ ❙: 웃} = Pause until non-trivial number of users, to merit the extra level of polish done by these features

v2.1

High-value features (Choose 1 or 2):

• Ability to search for URLs that did not download at all or did not download successfully (with an error) (#84) 👈(DF recommends)
• Sequential groups (#110)
  • Already have initial UI for this, behind the NewGroupDialog._SHOW_SEQUENTIAL_OPTION feature flag.

{❙ ❙: 웃} Easy to install for early adopters, on Linux:

• Fix install on Linux to not take 10+ minutes (#166)
• Add Fedora to CI (#93)

{❙ ❙: 웃} v3.0 - Welcome end users: Great first-run experience. Great documentation.

{❙ ❙: 웃} Easy to install for end users (Part 1):

• Binary install steps work on all platforms:
  • Can open Crystal binary on macOS, without special security bypass (#20)
    • 💰 Requires an Apple Developer ID certificate ($99/year)
  • Can open Crystal binary installer on Windows, without a publisher warning (#144)
    • 💰 Probably requires some kind of paid certificate from Microsoft
• —
• Easy to install/run on Mac, from disk image:
  • Mac Installation: Streamline process to install Crystal to Applications folder (#156)
• Easy to install/run on Windows, from installer:
  • Windows Installer: Offer to open Crystal after install. Offer to add desktop shortcut. (#155)

{❙ ❙: 웃} Easy to install for end users (Part 2):

• Easy to run on Mac, from Mac App Store:
  • Build/submit .app for Mac App Store (with M1 support too) (#42)
    • 💰 Requires an Apple Developer ID certificate ($99/year)
• Easy to run on Windows, from Windows Store:
  • [#] Investigate localization steps to make Crystal feel more like a native Windows app, if any...
  • Submit .exe to Windows App Store (#181)
    • 💰 Probably requires some kind of paid certificate from Microsoft
• Easy to run on Linux, from local package managers:
  • Submit .deb to Debian (& Ubuntu) package repository (#90)
  • Submit .rpm to Fedora (& Red Hat) package repository (#91)

Branding during first run:

• Better start dialog/window (#113)
  • Show Crystal logo and icon in first dialog
    • Maybe the first window is just an untitled project?
      • ✅ Already has branding.
      • 👉 May want to add an easy "Open" call-to-action button
• Main Window: Improve Crystal branding in lower-left corner (#99)

{❙ ❙: 웃} Easy to understand interface and concepts at a glance:

• Consider remove/minimize jargon "root" and "URL" from various places, like "New Root URL".
  • Change "New Root URL" → "New Root Page" or "New Page"
  • Change icon from (a bare ⚓) → (a 📄 with an ⚓ superimposed).
    • Pro: Aligns with icon for a regular "page" (📄)
    • Pro: Looks like an "anchored page"
    • Con: Cannot use Unicode icon to represent on macOS/Linux inside the [New Group > Source] dropdown.
  • ✖️ Change shortcut ⌘R (for "New Root URL") → «something else»
    • Maybe not. Not a lot of alternatives that are easy to type, especially with the left hand. ⌘D, ⌘E

Easy to create first URL:

• Better error messages for common download error types (#180)

{❙ ❙: 웃} Easy to create first group:

• Add Group: Warn when trying to create group using pattern with no wildcards (#77)

{❙ ❙: 웃} Easy to serve first URL:

• View: Long delay opening web browser (#149)
• Log Drawer: Animate open/close (#134)
  • Strengthen mental association between pressing View and the Log Drawer that appears

{❙ ❙: 웃} Easy/possible to export:

• Support export of project to filesystem (NEEDS ISSUE)
  • A user asked about this in #119
• Support export of project to static file server (like GitHub Pages) (#41)

Easy to quit:

• Respond to Quit event from OS correctly (#150)

Quality:

• Automated test coverage of core workflows:
  • Download Site Requiring Login:
    • test_can_download_and_serve_a_site_requiring_cookie_authentication (related to #68)
  • ... (probably more core workflows missing test coverage)

vFuture+0 Open Source - Other features

Any of the following items individually are major enough to warrant a release on their own (releaseworthy):

• Support dynamic download of site entirely through browser (#69)
• Support rewriting URLs on the frontend in JavaScript, which is necessary to archive more-modern sites that load many resources using JavaScript that cannot be fully statically analyzed to extract URLs in advance. 🧠 (#8)

vFuture+0 Commercial - Export/distribution of downloaded sites

...as an advanced contribution-required feature

• Support export of downloaded site to plain files that can be deployed to a static file server, either to a custom domain (best) or to a subdirectory within an existing domain (worse, because some URLs may not be as straightforward to rewrite). (#41)
  • Does round out final piece of end-to-end preservation lifecycle: capture, retain, & share
  • Does enable removing real xkcd from quickstart instructions.

Separate from the release of Crystal itself:

• Do use the export feature to export & host various notable sites I've archived that have fallen off the internet:
  • TMFFA -> tmffa.daarchive.net (#60)
  • PRFI -> rakhal.daarchive.net (#61)
  • (??) Apple Support Area -> ???.daarchive.net
    • ⚖️ Has special legal considerations: Would Apple disapprove of distributing the old files from this area?
  • –––
  • xkcd -> xkcd.daarchive.net
    • 🎗 Needs special robots.txt to prevent indexing, because original site still online
  • (??) OtakuWorld + The Big KiSS Page + Jenniverse -> (multiple!)
    • 🎗 Needs special robots.txt to prevent indexing, because original site still online
    • –––
    • -> *.otakuworld.daarchive.net
    • -> jenniverse.daarchive.net
    • -> *.unicornjelly.daarchive.net
    • -> pasteldefender.daarchive.net
    • -> transsexual.daarchive.net
    • -> kamishibai.daarchive.net

vFuture+1 Commerical - Support distributed crawling

...as an advanced contribution-required feature

• Distributed web crawling: Separate out crawling logic such that a centralized "orchestrator" can direct a remote fleet of "crawlers" to quickly capture a large domain that is scheduled to fall off the public internet soon (ex: Geocities, DeviantArt Portfolios, etc).