In order to make sure ansible has access to the boxes, I needed to remove the following line form my inventory file. Also I had to comment out the known_hosts setting.
I had the line in all:vars section
[all:vars]
ansible_ssh_private_key_file = ~/.ssh/id_rsa
After removing it, the playbook action could connect.