Use with VM yet to be created and without local docker daemon

[Real-Gecko]

I want to have a project that first creates some VMs and then this provider kicks in and provides some services that'll be running on those VMs.
How can make it work?
If I setup provider like this:

provider "docker"{}

I get an error:
Error: Error pinging Docker server: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
If I set it to:

provider "docker" {
  host = incus_instance.swarm_nodes[0].ipv4_address
}

I get Error: Error initializing Docker client: unable to parse docker host
Is there a way to skip initialization maybe?