dbt-spark: How to achieve spark parallel execution?

[UdroiuRuben]

Hello,

I’m trying to run three independent models in parallel using the following command: dbt run --select model_1 model_2 model_3

However, in the Spark UI, it appears that the models are being executed sequentially instead of in parallel.

Is it possible to achieve Spark jobs parallelism? Do you have any suggestions?

My profiles.yml configuration is as follows:

dbt_profile:
  target: dev
  outputs:
    dev:
      type: spark
      method: livy
      schema: data_pyspark_module_dev
      host: <>
      user: "{{ env_var('DBT_ENV_USERNAME') }}"
      password: "{{ env_var('DBT_ENV_SECRET_PASSWORD') }}"
      threads: 3
      livy_session_parameters:
        spark.app.name: "test_dbt_parallel"
        spark.scheduler.pool: "production"
        spark.scheduler.allocation.file: "hdfs:///applications/dependencies/fairScheduler.xml"
        spark.scheduler.mode: "FAIR"
        spark.driver.memory: "3000m"
        spark.executor.memory: "10000m"
        spark.executor.cores: "3"
        spark.executor.instances: "3"
        spark.yarn.queue: "root.jupyter"