The future of the History Project, a path forward

[wjdp]

This design document is going to be quite broad, proposing a way forward for the History Project without deep specifics on how each part will be designed. If accepted more docs will be needed on each part.

Problem

The process of building the History Project has become problematic as the code-base has aged.

• We rely on old versions of Ruby, Nodejs and their dependencies
• We use legacy package managers, e.g. Bower
• Getting the project building on a modern OS is near impossible, requiring a specific docker image with older versions of tools
• (Adjacent) Code and content is mixed in the same repository along with issue tracking and pull requests. When there is activity it's rather noisy.

Consequences

• There's a risk of this situation getting worse, where publishing the site becomes impossible
• Editors & contributors are unlikely to preview changes before merging
• Enhancements to the existing site are risky and more generally have stalled
• New contributors will be put off by the complexity and legacy of the project

Context

• The History Project is built from a single repository hosted in GitHub.
• Content is stored in Markdown files with YAML frontmatter
• Jekyll, a static site builder, is the core of the build process to turn this into a published website
• A significant amount of custom Ruby plugins extend Jekyll to our needs. Jekyll is built for content management of pages, we've turned it into a database.
  • Our content is structured data. The plugins build out that structured data into cross-linked pages and generated pages.
    • For example cast and crew in shows build out an in-memory database of people used to generate pages per person with a list of shows they were in.
  • These plugins patch and extend core Jekyll functionality. Any upgrade to Jekyll usually requires significant upgrades to these plugins.
    • Historically been the main blocker to keeping Jekyll updated
• Our dependencies include Ruby and JavaScript packages, generally the versions we are using are from ~2018
• We utilise several legacy frontend technologies
  • CoffeeScript is used for a significant amount of frontend code
  • Bower, a package manager has been in maintenance mode since ~2017
  • Libraries, for example:
    • fGrid, a styling library that hasn't been updated since 2015, yes that's my fault!
    • gulp v3 with lots of extensions
    • turbolinks
    • &c
• The build system is fragile and requires very specific versions of packages to work
  • Currently the local dev build is broken. It uses Docker to support the legacy Ruby and Node.js environment required. The fix (Attempt to fix the local dev environment #1849) seems to need other upgrades so is too risky to merge.
• The build in CI is working and is somewhat reliable… for now. It has broken in the past and there is a risk of it breaking in the future.
• A lambda function ntbotPostIssue is the only API endpoint in use. It's a single lambda function that creates GitHub issues from a web form submission.

Goals & Non-goals

Goals

• Provide a sustainable path forward for the History Project
• Lay out the major parts of a replacement build system
• Demonstrate a proof-of-concept system

Non-goals

• To design the individual parts of the replacement system

Design

Goals of the design

• To separate out major components into independent parts
  • Work on and deployment of one part should not impact other parts
• The 'blast radius' of each part should be reduced
  • e.g. the frontend should not need to be rebuilt to deploy new content
  • Replacement of a part should be possible without impacting other parts
    • e.g. if our frontend goes to shit after a few years we can build a new one without replacing the API build system
• To retain the resilience of the current content model
  • We're nearing our 100th anniversary, will the historical record still be around in another 100 years?
  • While the software we build to present the content may not last long, the raw content itself should be accessible and readable in perpetuity.
  • Content stored as Markdown with YAML in a Git repo has strong resiliency

Structure

I propose splitting the project into several major parts:

• A content repository
• A build service repository
• A frontend repository
• A backend service repository

Mapping that structure to what we have today

• The history-project repo will be relegated to only content
  • The structure will remain the same for now so it's backwards compatible with the existing build system
• A new API generator will be built to turn the content into a JSON API for consumption by a frontend (along with anything else)
  • A JSON API is compatible with pretty much anything we want to build the frontend with
• A frontend will be built that consumes this API
• Optionally a backend service will be built to take over from ntbotPostIssue and potentially be extended with additional functions such as alumni membership.

Proof of concept

• API generator https://github.com/newtheatre/nthp-api
  • This has been plugged into the content repo for quite some time
  • It's a Python project that loads the content repo into an in-memory database while making relations and then dumps out a static API of JSON files
  • You can preview the current state at https://elements-demo.stoplight.io/?spec=https://nthp-api.wjdp.uk/v1/master/openapi.json
  • Provides fast and accurate feedback for syntax errors in content files
• Frontend https://github.com/newtheatre/nthp-web
  • Preview it here https://nthp-web.pages.dev/
  • A very rough frontend, to validate the API POC is usable to generate a site.
• Backend service https://github.com/newtheatre/lumina
  • This I wouldn't give too much thought to at the moment. Has been more of an experiment in what could be possible.
  • Maintains a database of alumni members and tries to meld that with submissions to the history site. May fix the problem we have with web submissions being anonymous. Adds a login function via email.

Issues

• We're no longer building a static web site that simply pulls HTML from the server. The site becomes an app that consumes an API and builds the presentation in the users browser.
  • More fragile than pre-baked HTML
  • Buut much faster to build, we're able to do branch builds again and long build times are eliminated
  • With the maturity of current frontend frameworks compared to a few years ago this should last a while
  • The control against the frontend becoming outdated is the separation of the API build from the frontend repos

Alternatives

• We could retain the build system in a single repo
  • Pros
    • A single repo is easier to deal with, a single clone gets you everything to run the thing locally
    • Dependence between versions of parts is much easier to deal with as version control keeps changes in sync
  • Cons
    • The 'blast radius' issue isn't fixed, plus maintaining the old system while building the new is very tricky
    • The system will naturally become very coupled, meaning we'll end up in the same state as today again
• We could improve the current build system
  • Pros
    • Avoidance of bringing anything new in
  • Cons
    • The 'hacky Jekyll' system we have is high maintenance. It worked as I had intimate knowledge of the project ~10 years ago. No-one has that now.
    • We end up in the exact same situation in a few years
    • Hacking on top of Jekyll isn't what it's designed for, we'll be limited with introducing new features
• Move content into a traditional database and have an application host the site rather than having a build system
  • Pros
    • No need for a build system
    • Content editing and publishing could be made much faster
  • Cons
    • Complete loss of version control, we gain so much from Git and having an immutable history of the project
    • Real danger of data loss, a database requires we manage the backups and avoid corruption. A widely published git repo will outlive us. Using this method a copy of the project has been archived in a mountain.

Next steps

1. Discuss, amend, and agree on this proposal
2. Investigate the API generator prototype
3. Write up, discuss, and agree on a proposal for the API generator
4. Consider options for the frontend, lather/rinse/repeat process again there
5. Build out the API generator & the new frontend
6. Consider if we need a backend yet, perhaps do something about that
7. Launch the new history project!
8. Deprecate the old build system and clean up the content repo

[BenWoodford76]

Thanks for starting this discussion @wjdp, I know you spent a lot of time and effort in getting the site up and running in the first place and maintaining the back and front ends.

I shall preface the rest of my comments with the caveat that I am a content editor mainly, and have very little front or back end development experience.

I agree that the current method of build is unsustainable, with all the depreciated dependencies and custom built plugins that the site depends on its frankly surprising (and a testament to the progrmaming skill of those behind it) it hasnt broken more often.

I dont know how much of the current situation is simply down to the fact that those of us who were around at the beginning of this project, and who were all students at the time, simply dont have the resources of time to spend regularly updating things as weve moved into the adult world. I also think we've either not capitalised on or failed to bring in 'fresh blood' so to speak of current members with an interest in keeping this site up and running.

That being said, I am personally leaning towards the option you have outlined, where different sections of the site are split into different repos where modification can be made to one without having to drastically reprogram the others.

However we do decide to go forward with this, I think we need to do our utmost to include those of the current mmbership who want to be involved to join in the discussion.

May have more to say at a later time, but that's my initial thoughts on the matter.