chore: test parallelization of library generation

[JoeWang1127]

Run multiple docker containers in parallel

Build docker image

# In main branch of sdk-platform-java
git pull
DOCKER_BUILDKIT=1 time docker build \
  --file .cloudbuild/library_generation/library_generation.Dockerfile \
  -t local:latest \
  --target final .

Prepare API definitions

# In master branch of googleapis
git checkout 14e5759e6036ea42178a7e9cf2490d72cb6813a2

Run script

# In main branch of google-cloud-java
git checkout chore/one-per-process
./parallel-run.sh

Explanation

The script will start n docker run command running in the background, each command will generate 1/n libraries in one config.

Log of each execution will be sent to logs/${num}.log in the current working directory.

Note that you may need to change the path to API definitions:

# Change the path to googleapis in your environment.
-v "$(pwd)/../apis:${workspace_name}/apis" \

How to know the process exits

Run this commend in a separate console

ps -ef | grep docker

time spent in parallel generation (cloudtop)

7 process, ~30 libraries in each process

real 21m13.705s
user 0m0.164s
sys 0m0.279s

real 24m37.660s
user 0m0.188s
sys 0m0.242s

real 27m14.798s
user 0m0.173s
sys 0m0.314s

real 29m17.713s
user 0m0.225s
sys 0m0.287s

real 32m43.476s
user 0m0.261s
sys 0m0.274s

real 34m16.035s
user 0m0.180s
sys 0m0.311s

real 36m31.636s
user 0m0.263s
sys 0m0.236s

10 process, 20 libraries in each process

real 21m12.750s
user 0m0.192s
sys 0m0.148s

real 23m58.714s
user 0m0.175s
sys 0m0.161s

real 24m38.005s
user 0m0.205s
sys 0m0.192s

real 26m54.265s
user 0m0.242s
sys 0m0.183s

real 27m56.942s
user 0m0.152s
sys 0m0.230s

real 32m48.548s
user 0m0.252s
sys 0m0.249s

real 32m52.835s
user 0m0.167s
sys 0m0.241s

real 35m41.021s
user 0m0.162s
sys 0m0.256s

real 37m18.957s
user 0m0.194s
sys 0m0.226s

real 38m41.068s
user 0m0.187s
sys 0m0.229s

20 process, 10 libraries in each process

real 18m40.931s
user 0m0.079s
sys 0m0.142s

real 20m49.150s
user 0m0.145s
sys 0m0.096s

real 21m33.018s
user 0m0.104s
sys 0m0.134s

real 22m8.657s
user 0m0.115s
sys 0m0.118s

real 23m26.363s
user 0m0.150s
sys 0m0.125s

real 23m41.904s
user 0m0.135s
sys 0m0.122s

real 24m48.153s
user 0m0.145s
sys 0m0.117s

real 25m8.004s
user 0m0.155s
sys 0m0.138s

real 27m50.701s
user 0m0.129s
sys 0m0.134s

real 29m15.844s
user 0m0.160s
sys 0m0.135s

real 30m6.565s
user 0m0.124s
sys 0m0.160s

real 30m22.344s
user 0m0.151s
sys 0m0.127s

real 30m57.712s
user 0m0.175s
sys 0m0.121s

real 31m44.277s
user 0m0.189s
sys 0m0.112s

real 32m8.215s
user 0m0.169s
sys 0m0.179s

real 32m22.352s
user 0m0.168s
sys 0m0.151s

real 34m51.932s
user 0m0.199s
sys 0m0.107s

real 35m20.476s
user 0m0.165s
sys 0m0.157s

real 36m44.158s
user 0m0.181s
sys 0m0.141s

real 37m17.364s
user 0m0.211s
sys 0m0.134s

40 process, 5 libraries in each process

real 16m15.046s
user 0m0.083s
sys 0m0.141s

real 16m49.980s
user 0m0.097s
sys 0m0.127s

real 17m1.309s
user 0m0.073s
sys 0m0.132s

real 17m55.847s
user 0m0.109s
sys 0m0.256s

real 18m42.726s
user 0m0.080s
sys 0m0.192s

real 21m6.307s
user 0m0.125s
sys 0m0.281s

real 22m47.112s
user 0m0.129s
sys 0m0.192s

real 22m49.006s
user 0m0.094s
sys 0m0.189s

real 23m21.315s
user 0m0.097s
sys 0m0.191s

real 24m38.806s
user 0m0.150s
sys 0m0.182s

real 25m31.867s
user 0m0.378s
sys 0m0.305s

real 25m31.896s
user 0m0.119s
sys 0m0.289s

real 25m33.633s
user 0m0.139s
sys 0m0.203s

real 25m51.540s
user 0m0.127s
sys 0m0.196s

real 25m58.760s
user 0m0.148s
sys 0m0.210s

real 28m23.565s
user 0m0.112s
sys 0m0.228s

real 29m26.143s
user 0m0.137s
sys 0m0.238s

real 30m51.697s
user 0m0.102s
sys 0m0.239s

real 30m59.837s
user 0m0.109s
sys 0m0.279s

real 31m3.054s
user 0m0.137s
sys 0m0.172s

real 31m5.769s
user 0m0.128s
sys 0m0.202s

real 31m36.027s
user 0m0.123s
sys 0m0.246s

real 32m0.400s
user 0m0.156s
sys 0m0.245s

real 32m21.159s
user 0m0.123s
sys 0m0.230s

real 33m9.436s
user 0m0.167s
sys 0m0.220s

real 33m21.898s
user 0m0.161s
sys 0m0.222s

real 33m26.184s
user 0m0.179s
sys 0m0.229s

real 33m40.365s
user 0m0.141s
sys 0m0.298s

real 34m24.851s
user 0m0.120s
sys 0m0.244s

real 34m33.883s
user 0m0.159s
sys 0m0.178s

real 35m5.061s
user 0m0.131s
sys 0m0.253s

real 35m16.473s
user 0m0.115s
sys 0m0.253s

real 35m19.436s
user 0m0.112s
sys 0m0.226s

real 36m37.298s
user 0m0.093s
sys 0m0.272s

real 37m6.340s
user 0m0.204s
sys 0m0.180s

real 37m8.215s
user 0m0.158s
sys 0m0.221s

real 39m37.465s
user 0m0.164s
sys 0m0.201s

real 40m54.129s
user 0m0.145s
sys 0m0.253s

real 41m3.534s
user 0m0.198s
sys 0m0.202s

real 41m43.398s
user 0m0.206s
sys 0m0.220s