Speed up mirror-generation through parallelization

[Vinai]

@tdgroot had the idea to enable parallelized builds by making a copy of the git repo for each tag.
This would allow each tag to be processed in parallel, which would make a pretty big difference for the mirror-magento2 repo.

[Vinai]

A bit of complexity is added because for mage-os releases, the working copies are used to push the release tags back to GitHub.
So either that process will have to be adjusted to use the changed directory structure, or we limit the parallelization to the mirror generation where it is needed. The release build process would not benefit from it because it doesn't operate on multiple tags.

[fballiano]

I've done some tests in #221, but the problem I ended up having is that, at the end of the day, there a a huge amount of git operations on the same repos and they're all competing agains each other.

So from my experiment I couldn't get any significant improvement in the overall speed of the process, and the minor improvements probably don't justify the added complexity.

Then I thought we could just git worktree to checkout a worktree for every tag of the repos, so that the rest of the processes could run in parallel, but the filesystem exploded and I couldn't even finish the testing. If we had a powerful MVME based server, probably we could make it work, but we don't have that and github workers are not the place for such an intensive operation.

IMHO it would be better to be able to build a single version and never rebuild previously distributed packages.

[rhoerr]

IMHO it would be better to be able to build a single version and never rebuild previously distributed packages.

What would it take to make that happen? Check for package existence, I guess? Maybe some additional step to check the integrity of those existing packages.