feat: distributed compilation support

[kcelestinomaria]

1. We should enable multi-core CPU compilation using Python multiprocessing or Dask, and

2. Plan architecture for cluster-based distributed compilation with Ray or Spark.