Split monorepo up into several repos

[aslakhellesoy]

Is your feature request related to a problem? Please describe.

There are several problems with the cucumber/common monorepo:

• Building all the 50+ packages in the repo takes ~1h (both locally and in CI)
• The build system is complex, brittle and hard to maintain
• Newcomers find the size of the repo and complexity of the build process intimidating and hard to navigate

Describe the solution you'd like

I want to split cucumber/common up into multiple repos:

• Polyglot repos (with java, javascript, ruby etc subdirectories)
  • cucumber/cucumber-expressions
  • cucumber/tag-expressions
  • cucumber/gherkin
  • etc.
• Single implementation repos
  • cucumber/language-service
  • cucumber/language-server
  • cucumber/vscode
  • cucumber/monaco
  • cucumber/react
  • etc.

The current build system has complex functionality that we'd have to replace:

• Consistent build configs
  • Extract to cucumber/eslint, cucumber/tsconfig etc. git repos.
• Release scripts (Update version numbers in descriptors/changelogs, package and publish)
  • Extract to bash scripts in a cucumber/release git repo and use from Docker when doing a release
• Update version numbers in dependent libraries after a release
  • Rely on WhiteSource Renovate to do this

Also see discussion in Slack

Describe alternatives you've considered

We could probably push ahead with #1720 and make the current monorepo serial build run in a few seconds (by leveraging a cache in the cloud), but the build process would still be complex and brittle. Newcomers would still be intimidated by size and complexity of this huge repo.

Additional context

In 2015 the Cucumber implementations had diverged and behaved inconsistently. Each release made them more inconsistent. To mitigate this we decided to bring all the Gherkin implementations into one repository, using a shared acceptance test suite.

This worked well, so we continued with the same approach for new libraries such as Cucumber Expressions and Tag Expressions - in the same repo.

Building and in particular releasing libraries in 10 or so languages is complicated, so we built an "orchestration" build system with Make that makes the build process consistent across the increasing number of libraries.

Fast forward six years, and we have a build system with fangs, tentacles and worts. The build system wasn't designed with parallelism in mind, which is why it takes 1h.

TODO

• cucumber-expressions
• tag-expressions
• ...

[mpkorstanje]

I'll move datatable over to cucumber-jvm.

[aurelien-reeves]

Maybe we could do that in conjunction with #1614?

[aslakhellesoy]

While it is possible to bring commit history when moving from one repo to another, I suggest we don't do it because it's tedious to do. People who need to look at the history will find it in this repo.

[jamietanna]

We can retain the history for the new repos, for just the subtree that's being imported. I'd much prefer, as a consumer of Cucumber, to be able to view the history in the new repo, rather than jumping around.

I've done it before using Option 3 in https://stackoverflow.com/a/30386041 and it works nicely.

[aurelien-reeves]

Thanks for the info @jamietanna!

If we have the possibility to keep the history, that would be great indeed!

[aslakhellesoy]

Ok, I'm sure we can do that :-)

[aslakhellesoy]

I'm proposing we start by creating the following new repositories:

• https://github.com/cucumber/editor (pure typescript repo)
  • packages
    • language-server
    • language-service
    • monaco
    • suggest
    • vscode
    • prosemirror (from Markdown editor #1618)
• https://github.com/cucumber/tag-expressions
  • go
  • java
  • javascript
  • python
  • ruby
• https://github.com/cucumber/cucumber-expressions
  • go
  • java
  • javascript
  • ruby
  • testdata
• https://github.com/cucumber/cucumber-jvm
  • datatable I propose we combine the matchers and datatable modules into just datatable
• https://github.com/cucumber/messages (I'm aware we want to move the GherkinDocument messages into Gherkin - can happen after the move)
  • dotnet
  • elixir
  • go
  • java
  • javascript
  • jsonschema
  • perl
  • ruby
• https://github.com/cucumber/gherkin-utils
  • java
  • javascript
• https://github.com/cucumber/react (typescript-only repository)

This will hollow out about half of the monorepo. Eventually I would like to have everything moved out so we can retire the monorepo, but let's start with this.

[aurelien-reeves]

cucumber-expressions/test-data may be tricky to move.
Do we have a plan for shared test-data, CCK, and related?

Beside that, looks good 👍

[aslakhellesoy]

cucumber-expressions/test-data may be tricky to move

Why? I'm proposing we move it along with all the implementations so the directory structure will be like this:

.
├── go
├── java
├── javascript
├── ruby
└── testdata

[aurelien-reeves]

Oh, testdata here are not synced from another package 😅

Sorry for that. So yes, looks good 👍

[aslakhellesoy]

Do we have a plan for shared test-data, CCK, and related?

We have two kinds:

• Test data that is not used outside a module, such as cucumber-expressions/testdata
  • This will just be moved along with the source code
• Test data that is used outside the module where it lives, such as gherkin/testdata and compatibility-kit/javascript/features/**/.ndjson
  • When these move to new cucumber/compatibility-kit and cucumber/gherkin repositories, other modules that need them could access the files via native language modules (maven, npm, gem etc - assuming we bundle them inside). Alternatively the builds could download them from GitHub with a git submodule or fetching a tarball.

[mpkorstanje]

@aslakhellesoy I'll move datatable to cucumber-jvm.

[mpkorstanje]

git remote add common git@github.com:cucumber/common.git
git fetch common
git checkout -b merge-datatable common/main 
git filter-branch --subdirectory-filter datatable
git merge origin/main --allow-unrelated-histories 
git push 

Looks like this worked for me. But notice the big disclaimer. Probably good to follow it.