dbt pyspark model in wsl local environment

[rarpal]

I have setup pyspark in wsl (windows 11) with hive support and metastore enabled and working with spark-sql which is working as expected. I have now installed dbt-spark with dbt-spark[session] adaptor. What I want is to be able to create python dbt models with the following configuration in my profiles.yml:

local_spark:
  target: dev
  outputs:
    dev:
      type: spark
      host: local
      method: session
      schema: default
      file_format: delta

As you can see I am trying to run my python model using an active local spark session (method:session).
I run my python model via a entry point python script which creates a PySpark session before calling dbtRunner dbt.invoke method as shown below:

    # Initialize the dbtRunner
    dbt = dbtRunner()

    # Define the dbt command to run, for example: `dbt run --models my_model`
    cli_args = ["run", "--select", "stage", "--project-dir", "<my project folder>", "--profiles-dir", "<my profile folder>"]

    # Use the invoke method to run the command
    result = dbt.invoke(cli_args)

However when I run this I get the following error:
Databricks cluster_id is required for all_purpose_cluster submission method with running with notebook.

Even though I am not using the dbt-databricks adaptor and have requested the method:session

Does anyone know what the issue is here and If I am missing some config ?