How does laravel achieve it's illuminate subtree splits?

[hirasso]

Dear Laravel community!

I recently discovered that Laravel provides subtree splits of laravel/framework under https://github.com/illuminate, all marked as [READ-ONLY]. I'm assuming development takes place in laravel/framework and all those illuminate packages are being synced automatically somehow? I would be very grateful if someone could shed some light on how that is done. Maybe there even exists a blog article about the setup somewhere out there?

Thanks for any insight 🙏

[hirasso]

Looks like the tool https://github.com/splitsh/lite is used for creating the splits. Relevant files:

• https://github.com/laravel/framework/blob/11.x/bin/split.sh performs the split
• https://github.com/laravel/framework/blob/11.x/bin/release.sh tags the repos automatically
• the release.sh script seems to be executed in some dedicated 'splitter server':

framework/.github/workflows/releases.yml

Lines 32 to 42 in 48246da

	- name: SSH into splitter server
	uses: appleboy/ssh-action@master
	with:
	host: 104.248.56.26
	username: forge
	key: ${{ secrets.SSH_PRIVATE_KEY_SPLITTER }}
	script: |
	cd laravel-${{ github.ref_name }}
	git pull origin ${{ github.ref_name }}
	bash ./bin/release.sh v${{ steps.version.outputs.version }}
	script_stop: true

Remaining Questions:

• why exactly does the release action require a remote server to run the release split?
• how is it possible to run the local script ./bin/release.sh on that remote ssh server? I'm confused 😄

[Sophist-UK]

I can answer the question how is it possible to run the local script ./bin/release.sh, but I can't answer other questions I consequently have. The script runs on the remote server at 104.248.56.26, and it:

1. After SSH login, the working directory is presumably the user forge's home directory.

2. It changes to a subdirectory of the home directory which (I think) is the release tag. Q: How does this directory get created because the script assumes that it already exists??!!

3. It does a git pull. Q: How does it know which repository to pull? Potential A: Because the now CWD (tagged subdirectory of the forge home directory) is the root of a predefined git clone of this repository on that server.

4. It then runs the script {gitroot}/bin/release.sh that has just been pulled. Existing Q: how is it possible to run the local script ./bin/release.sh on that remote ssh server? A: It is the same local script that is now on the remote server as part of a git clone.

5. The other thing I am unclear about is... Q: why they use a private server for this when you could presumably do it as a normal github action bash script?

So I seem to have raised more questions than I was able to answer.

[mbabker]

The other thing I am unclear about is... Q: why they use a private server for this when you could presumably do it as a normal github action bash script?

The splitsh-lite package creates a local database that helps it run more efficiently by keeping track of commits/trees/whatever it's already processed. Spinning up a new container for every use like https://github.com/marketplace/actions/splitsh-lite-action or a hand-crafted script would do is going to lose that database. And for big trees like Symfony's or Laravel's where splits are happening pretty regularly (IIRC every push to the Symfony repo triggers the splitter), that's not very efficient.

[Sophist-UK]

That makes sense - but for my own project I am only going to do a split on a release and not on commits.

[mbabker]

The frequency doesn't really matter as it's all still based on the size of the tree, the advantage to the cache is that splitsh-lite is going to know "I've already processed up to HEAD~3 so I only need to process these last 3 commits during this run". If it's a smaller monorepo with not too many split paths, then just doing it all in a GItHub action might be OK for you (I've been doing splits on https://github.com/BabDev/Pagerfanta for a few years now from my laptop (I've just been too lazy to hook it up to CI 😅) and my .git/splitsh.db is a little over 500KB). But if it's a larger repo with a lot of activity or has a lot of split paths, then having to rebuild that database from scratch every time might not be very efficient.

Both approaches will work just fine and produce a consistent result, but there are going to be tradeoffs with each one at some point. That's really the only feedback I have to offer here based on my own experiences.

[Sophist-UK]

@mbabker Michael, Thanks for taking the time to explain further.

[hirasso]

Hunted down the relevant PRs/commits:

• the release action was introduced just recently by @driesvints in [10.x] Release action #49838
• the splitter was introduced by @taylorotwell in 2018: 80ef6d7

It would be amazing if one of you could provide some insight on how you did this and what were your decisions/findings that led to the implemented solution 🙏

A bit of background why I'm so interested: I'm a maintainer of https://github.com/swup/swup and we'd like to move from a manyrepo to a monorepo (one core, many core plugins). Splitting the plugins back out into dedicated read-only repos for each would hugely benefit their visibility on GitHub, as opposed to hiding them deep in our monorepo.

[Sophist-UK]

I'm a maintainer of https://github.com/swup/swup and we'd like to move from a manyrepo to a monorepo (one core, many core plugins). Splitting the plugins back out into dedicated read-only repos for each would hugely benefit their visibility on GitHub, as opposed to hiding them deep in our monorepo.

Ditto for a major open source project I have in mind which will be very plugin oriented and have huge numbers of plugins.