install_prereq.sh: Build more of the prereqs in parallel.

Speed up the bottleneck builds in the CI by building in parallel
several of the most time-consuming to build libraries. The containers
used by the GitHub CI have 4 cores, and the prereq install phase
can be sped up non-trivially by doing this.

Benchmarking the total time to run install_prereq.sh (on Debian
systems where all the packages have already been installed),
compiling libdiscord, libetpan, and libopenarc in parallel cuts
the script execution time almost in half. Even though using -j$(nproc)
across 3 scripts theoretically results in 12 threads running on
a system with only 4 cores, this still yielded better performance
than just running in parallel alone, so all three scripts now use
-j$(nproc) in addition to being run in parallel. This yields
greater user time but the lowest wall clock time overall.

All systems running any of the scripts alone will benefit from
the -j$(nproc), but in practice, all 3 libraries are only supported
on Debian-based distros, so only Debian-based systems will realize
the overall speedup.

(AWS t3.xlarge, 4 vCPU, 16 GB - same specs as a GitHub CI runner):
Serial, without -j$(nproc):
real    2m32.726s
user    2m8.945s
sys     0m27.937s

Serial, with -j$(nproc):
real    1m44.368s
user    2m27.967s
sys     0m28.350s

Parallel, without -j$(nproc):
real    1m50.111s
user    2m15.002s
sys     0m26.602s

Parallel, with -j$(nproc):
real    1m12.970s
user    2m40.010s
sys     0m27.909s

(AWS t3.small, 2 vCPU, 2 GB of RAM):
Serial compilation, without -j$(nproc, except in libdiscord.sh:
real    2m27.488s
user    2m5.345s
sys     0m20.269s

Parallel, without -j$(nproc, except in libdiscord.sh:
real    1m53.429s
user    2m16.722s
sys     0m20.528s

Parallel, and with -j$(nproc):
real    1m41.834s
user    2m41.015s
sys     0m24.475s

Author: InterLinked1

scripts/install_prereq.sh

@@ -196,6 +196,11 @@ PACKAGES_DEBIAN="$PACKAGES_DEBIAN bc"
 PACKAGES_DEBIAN="$PACKAGES_DEBIAN html2text"
 # INSTALL_LIBETPAN=0 on Fedora-based systems, so don't bother installing html2text on those.
 
+# doxygen only: env var required to enable
+if [ "$INCLUDE_DOXYGEN" = "1" ]; then
+	PACKAGES_DEBIAN="$PACKAGES_DEBIAN doxygen graphviz"
+fi
+
 # Required only for tests
 if [ "$1" = "1" ]; then
 	# mariadb-server is required for some of the test modules (test_auth_mysql, test_irc_chanserv)
@@ -306,34 +311,35 @@ scripts/libwss.sh
 # mod_slack (also depends on libwss)
 scripts/libslackrtm.sh
 
-# libdiscord (mod_discord)
-if [ "$INSTALL_LIBDISCORD" != "0" ]; then
-	# Makefile doesn't add the proper LDFLAGS on BSD and doesn't seem to obey LDFLAGS when passed in...
-	scripts/libdiscord.sh
-fi
-
-# libetpan fails to build successfully on Fedora-based distros,
-# so need to be able to skip that for now using an env var.
-if [ "$INSTALL_LIBETPAN" != "0" ]; then
-	# libetpan (mod_webmail): the package no longer suffices, since we patch the source.
-	#PACKAGES_DEBIAN="$PACKAGES_DEBIAN libetpan-dev"
-	scripts/libetpan.sh
-
-	# evergreen (door_evergreen)
-	scripts/evergreen.sh
+if [ "$INSTALL_LIBDISCORD" != "0" ] && [ "$INSTALL_LIBETPAN" != "0" ] && [ "$INSTALL_LIBOPENARC" != "0" ]; then
+	# On machines with many CPU cores, take advantage of the parallelism available by building several libraries at once
+	printf "Building libdiscord, libetpan, and libopenarc in parallel\n"
+	(trap 'kill 0' INT; scripts/libdiscord.sh & (scripts/libetpan.sh && scripts/evergreen.sh) & scripts/libopenarc.sh & wait)
 else
-	printf "Skipping libetpan install (INSTALL_LIBETPAN=%s)\n" "$INSTALL_LIBETPAN"
-fi
+	# libdiscord (mod_discord)
+	if [ "$INSTALL_LIBDISCORD" != "0" ]; then
+		# Makefile doesn't add the proper LDFLAGS on BSD and doesn't seem to obey LDFLAGS when passed in...
+		scripts/libdiscord.sh
+	fi
 
-# mod_smtp_filter_arc
-# milter pre-req can be hard to satisfy, so can be disabled using an env var
-if [ "$INSTALL_LIBOPENARC" != "0" ]; then
-	scripts/libopenarc.sh
-else
-	printf "Skipping libopenarc install (INSTALL_LIBOPENARC=%s)\n" "$INSTALL_LIBOPENARC"
-fi
+	# libetpan fails to build successfully on Fedora-based distros,
+	# so need to be able to skip that for now using an env var.
+	if [ "$INSTALL_LIBETPAN" != "0" ]; then
+		# libetpan (mod_webmail): the package no longer suffices, since we patch the source.
+		#PACKAGES_DEBIAN="$PACKAGES_DEBIAN libetpan-dev"
+		scripts/libetpan.sh
 
-# doxygen only: env var required to enable
-if [ "$INCLUDE_DOXYGEN" = "1" ]; then
-	PACKAGES_DEBIAN="$PACKAGES_DEBIAN doxygen graphviz"
+		# evergreen (door_evergreen)
+		scripts/evergreen.sh
+	else
+		printf "Skipping libetpan install (INSTALL_LIBETPAN=%s)\n" "$INSTALL_LIBETPAN"
+	fi
+
+	# mod_smtp_filter_arc
+	# milter pre-req can be hard to satisfy, so can be disabled using an env var
+	if [ "$INSTALL_LIBOPENARC" != "0" ]; then
+		scripts/libopenarc.sh
+	else
+		printf "Skipping libopenarc install (INSTALL_LIBOPENARC=%s)\n" "$INSTALL_LIBOPENARC"
+	fi
 fi

scripts/libetpan.sh

@@ -30,5 +30,5 @@ wget "https://patch-diff.githubusercontent.com/raw/dinhvh/libetpan/pull/447.diff
 git apply 447.diff
 
 ./autogen.sh --with-poll
-make
+make -j$(nproc)
 make install

scripts/libopenarc.sh

@@ -11,5 +11,5 @@ else
 fi
 git checkout develop # master branch is behind
 # https://github.com/trusteddomainproject/OpenARC/issues/118
-aclocal && autoconf && autoreconf --install && automake --add-missing && ./configure && make all
+aclocal && autoconf && autoreconf --install && automake --add-missing && ./configure && make all -j$(nproc)
 make install