Updating many packages at once is not optimal

[gaborcsardi]

Because we might run into the six hour limit, or an error in the middle of deployment would throw away all work.

Ideally the update code in pkgdepends would update the repo periodically, after every (say) 30 minutes or 50 packages, whichever comes first?

That would have to be implemented in pkgdepends as a hook, probably, that is called after every package is installed and/or built.

[gaborcsardi]

Another way to do this would be to have a two-step procedure, one is to install the packages, and then in the second step we would build the binaries and deploy them.

We would put a ~5h time limit to the first step, and then the remaining ~1h is plenty for the reployment. We would run the deployment even if the first step timed out, or even if it failed.

This would still require a manual re-run of timed out builds to build the remaining packages, but that's probably OK, this should really only come up (currently) when populating an entirely new repository. Later, if we have more packages in the repos, we can refine this.

[gaborcsardi]

Another issue is a failure while adding the release assets:

ℹ Creating GH release cran/png 0.1-8.
Error in stop(http_error(resp)) : 
  Unprocessable Entity (WebDAV; RFC 4918) (HTTP 422).
Calls: <Anonymous> ... upload_releases -> <Anonymous> -> synchronize -> <Anonymous>
Execution halted
Error: Process completed with exit code 1.

Then the whole build is ruined. :(

[gaborcsardi]

The best would be to publish the release immediately after the package was installed. This could be another task that runs in a subprocess. We would need to make sure that we do not publish the releases too fast, as GH will throw an error.

In addition to this, we'd also need to update repository quite frequently, but it is unclear how that would work. We probably do not want 200 git pushes for updating 200 packages. OTOH maybe we do. I wonder how homebrew is doing this.