ODK should support parallel builds

[gouttegd]

We do not explicitly or officially support parallel builds (e.g., sh run.sh make -j 5 ...), and it is in fact unclear if our current workflows wouldn’t break in a parallel build.

However, there should be no reason why parallel builds could not be supported. If a workflow fails during a parallel build, that would just indicate that not all dependencies are properly listed – so it would in fact reveal a real issue with the workflow, not just with parallel build.

Not all ontologies may benefit from parallel builds. For large ontologies, the build process is more likely to be limited by memory than by CPU, and for those a parallel build would not help (it could make things worse, in fact).

But for ontologies that are small enough to be CPU-bound rather than memory-bound (or for people who have access to machines with enough memory that their builds are never memory-bound), being able to make parallel builds would be very nice.

[ptgolden]

Anecdotally, I've been using -j when building https://github.com/obophenotype/zebrafish-phenotype-ontology/ and haven't run into any problems.

I do believe the templated Makefile has an accurate dependency graph. The difficulty here would be that there is no way to ensure that the custom ont.Makefile does as well. Maybe just a documentation issue: "Run make in parallel to speed up CPU-bound processes, but note that it will act unpredictably without an accurate dependency graph in your custom Makefile."

[gouttegd]

Anecdotally, I've been using -j when building https://github.com/obophenotype/zebrafish-phenotype-ontology/ and haven't run into any problems.

Thanks, that’s very good to know.

The difficulty here would be that there is no way to ensure that the custom ont.Makefile does as well.

Good point, and indeed there’s not much we can do about that beyond pointing the issue in the documentation.

[ptgolden]

Not all ontologies may benefit from parallel builds. For large ontologies, the build process is more likely to be limited by memory than by CPU, and for those a parallel build would not help (it could make things worse, in fact).

I'll note that there's a third category where -j is quite helpful-- IO-bound tasks like mirror-%. (Well, it's IO-bound if you, like me, have a subpar Internet connection).

[twhetzel]

I have long wanted parallel builds for the mondo-ingest build. I believe I tried with -j, but think I ran into issues with the number of sparql queries.