DHCP domain?

[cdenneen]

Having a problem with oscar coming up and getting the domain from my DHCP server.

==> master: 2018-01-08 15:20:57,260 - [Notice]: Compiled catalog for master.example.com in environment production in 4.34 seconds

Then later during the puppet run it actually fails trying to resolve master.example.com

2018-01-08 15:26:01,081 - [Notice]: /Stage[main]/Puppet_enterprise::Console_services/Service[pe-console-services]/ensure: ensure changed 'stopped' to 'running'
2018-01-08 15:26:01,741 - [Error]: Failed to apply catalog: Could not connect to the Node Manager service at https://master.example.com:4433/classifier-api: #<SocketError: getaddrinfo: Name or service not known>
* /opt/puppetlabs/puppet/bin/puppet enterprise configure  --detailed-exitcodes --modulepath /opt/puppetlabs/server/data/enterprise/modules
* returned: 1

How can I avoid oscar from getting the domain from DHCP since it's using auto network?

Is there a way to configure oscar to use a dummy domain instead which will be added to the /etc/hosts entries as alias?

[cvquesty]

You can use the Vagrant plugin "vagrant-hosts" and utilize a stanza like such in your Vagrantfile:

master.vm.provision :hosts do |entries|
entries.add_host '192.168.0.5', ['master.puppet.vm', 'master']
end

[cdenneen]

@cvquesty I'm using oscar init-vms which creates the necessary configuration files. Wouldn't these modifications need to be put in there somehow since it's using auto_network?

I'm using vagrant-hosts and vagrant-auto_network along with pe_build and config_builder with oscar... so this should be a default oscar setup.

[cdenneen]

@Sharpie looks like same issue here that you helped solve so hoping to pick your brain... oscar-stack/oscar#54
I could probably stick master.example.com into the array as follows but what if I'm on a network that has domain set to foo.com now I would need to add master.foo.com:

---
roles:
  pe-puppet-master:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
        customize:
          - [modifyvm, !ruby/sym id, '--memory', 2048]
    provisioners:
      #- {type: hosts, sync_hosts: true}
      - type: hosts
        sync_hosts: true
        hosts:·
          - ['@vagrant_private_networks', ['@vagrant_hostnames', 'master.example.com']]
      - {type: pe_bootstrap, role: !ruby/sym master}

  pe-puppet-agent:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
    provisioners:
      - {type: hosts, sync_hosts: true}
      - {type: pe_agent, master: pe-puppet-master}

but I'm guessing this isn't the best way to handle this.
Is there anything that can be done to add FQDN to the vagrant-hosts entries?
Obviously if I do this offline without the DHCP domain getting in the way it would just have master and should work as expected but can't imagine always having to disconnect network in order to spin this up being a valid solution.

[cdenneen]

OK so updating the config with inserting the FQDN didn't work and actually isn't putting master in /etc/hosts on first:

---
roles:
  pe-puppet-master:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
        customize:
          - [modifyvm, !ruby/sym id, '--memory', 2048]
    provisioners:
      #- {type: hosts, sync_hosts: true}
      - type: hosts
        sync_hosts: true
        hosts:
          - ['@vagrant_private_networks', ['@vagrant_hostnames', 'master.example.com']]
      - {type: pe_bootstrap, role: !ruby/sym master}

  pe-puppet-agent:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
    provisioners:
      #- {type: hosts, sync_hosts: true}
      - type: hosts
        sync_hosts: true
        hosts:
          - ['@vagrant_private_networks', ['@vagrant_hostnames', 'master.example.com']]
      - {type: pe_agent, master: pe-puppet-master}

work/pe » vagrant ssh master
Last login: Mon Jan  8 17:45:24 2018 from 10.0.2.2
[vagrant@master ~]$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 master
10.20.1.2 master master.example.com
[vagrant@master ~]$ exit
logout
Connection to 127.0.0.1 closed.
work/pe » vagrant ssh first
Last login: Mon Jan  8 17:45:23 2018 from 10.0.2.2
[vagrant@first ~]$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 first
10.20.1.3 first master.example.com

==> first: curl -ksS --tlsv1 https://pe-puppet-master:8140/packages/current/install.bash -o pe_frictionless_installer.sh
==> first: bash pe_frictionless_installer.sh
    first: Running: inline script
    first: curl: (6) Could not resolve host: pe-puppet-master; Unknown error
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

So probably need to configure this differently.
Looking for it to look more like this:

work/pe » vagrant ssh master
Last login: Mon Jan  8 17:45:24 2018 from 10.0.2.2
[vagrant@master ~]$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 master master.example.com
[vagrant@master ~]$ exit
logout
Connection to 127.0.0.1 closed.
work/pe » vagrant ssh first
Last login: Mon Jan  8 17:45:23 2018 from 10.0.2.2
[vagrant@first ~]$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 first
10.20.1.2 pe-puppet-master # don't think the master and master.example.com are necessary since it appears to be using this alias which isn't being updated on the host with vagrant-hosts

[Sharpie]

@cdenneen Reversing the order of the array might work:

['master.example.com', '@vagrant_hostnames']

@vagrant_hostnames is just a magic string that expands out to an array of [VM name, VM hostname] --- as hostname -f/facter fqdn is sensitive to which value appears first on an /etc/hosts line. The first value is treated as the name assigned to the IP, returned by hostname, whereas the rest are "aliases".

One other thing that could be happening is that the VM name is master, which doesn't include a domain component. Unless there is a hostname: entry that sets a FQDN for that node, it could be falling through to whatever DHCP provides.

Most of the definitions I use have .puppetdbug.vlan set as the domain component, like:

---
vms:
  - name: master
    hostname: master.puppetdebug.vlan

Which might cause the hostname update applied by Vagrant post-boot to override DHCP since there is a domain component present in hostname.

[cdenneen]

@Sharpie why would reversing the array work? Would want master's ip in the agents hosts file.

I'll give the hostname definition a whirl to fix the FQDN, DHCP issue but it would be useful to maybe set a :domain = 'oscar.test' and have that apply to everything to avoid DHCP causing weird havoc?

[cdenneen]

@Sharpie

Okay so going back to the following roles.yaml:

---
roles:
  pe-puppet-master:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
        customize:
          - [modifyvm, !ruby/sym id, '--memory', 2048]
    provisioners:
      - {type: hosts, sync_hosts: true}
      # - type: hosts
      #   sync_hosts: true
      #   hosts:
      #     - ['@vagrant_private_networks', ['@vagrant_hostnames', 'master.example.com']] # this adds master.example.com to current hostname as alias... so on master and on first
      - {type: pe_bootstrap, role: !ruby/sym master}

  pe-puppet-agent:
    private_networks:
      - {ip: '0.0.0.0', auto_network: true}
    providers:
      - type: virtualbox
        linked_clone: true
    provisioners:
      - {type: hosts, sync_hosts: true}
      # - type: hosts
      #   sync_hosts: true
      #   hosts:
      #     - ['@vagrant_private_networks', ['@vagrant_hostnames', 'master.example.com']]
      - {type: pe_agent, master: pe-puppet-master}

And updating the vms.yaml:

---
vms:
- name: master
  hostname: master.puppetdebug.vlan
  box: puppetlabs/centos-7.2-64-nocm
  roles:
  - pe-puppet-master
- name: first
  hostname: first.puppetdebug.vlan
  box: puppetlabs/centos-7.2-64-nocm
  roles:
  - pe-puppet-agent

Fixes the master trying to resolve itself error but doesn't fix the client:

==> first: Running provisioner: hosts...
==> master: Updating hosts on: master
==> first: Running provisioner: pe_agent...
==> first: Running: set -e
==> first: curl -ksS --tlsv1 https://pe-puppet-master:8140/packages/current/install.bash -o pe_frictionless_installer.sh
==> first: bash pe_frictionless_installer.sh
    first: Running: inline script
    first: curl: (6) Could not resolve host: pe-puppet-master; Unknown error
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
work/pe » 

work/pe » vagrant ssh master
Last login: Tue Jan  9 21:25:42 2018 from 10.0.2.2
[vagrant@master ~]$ more /etc/hosts
127.0.0.1 localhost
127.0.1.1 master.puppetdebug.vlan master
10.20.1.3 first.puppetdebug.vlan first
10.20.1.2 master.puppetdebug.vlan master
[vagrant@master ~]$ exit
logout
Connection to 127.0.0.1 closed.
work/pe » vagrant ssh first
Last login: Tue Jan  9 21:25:42 2018 from 10.0.2.2
[vagrant@first ~]$ more /etc/hosts
127.0.0.1 localhost
127.0.1.1 first.puppetdebug.vlan first
10.20.1.3 first.puppetdebug.vlan first
10.20.1.2 master.puppetdebug.vlan master

So the question now is

1. how do I get pe-puppet-master as alias on the same line as the master line...

2. "why is it trying to use that endpoint and oscar doesn't add the alias by default?"

[cdenneen]

So I went and updated the name of the master in the vms.yaml to pe-puppet-master and I think this should have worked.
Seems firewalld might be causing issue?

==> first: curl -ksS --tlsv1 https://pe-puppet-master:8140/packages/current/install.bash -o pe_frictionless_installer.sh
==> first: bash pe_frictionless_installer.sh
    first: Running: inline script
    first: curl: (7) Failed connect to pe-puppet-master:8140; Connection refused
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

work/pe » vagrant ssh first
Last login: Tue Jan  9 21:59:42 2018 from 10.0.2.2
[vagrant@first ~]$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 first.puppetdebug.vlan first
10.20.1.2 first.puppetdebug.vlan first
10.20.1.6 master.puppetdebug.vlan master pe-puppet-master
[vagrant@first ~]$

---
vms:
- name: pe-puppet-master
  hostname: master.puppetdebug.vlan
  box: puppetlabs/centos-7.2-64-nocm
  roles:
  - pe-puppet-master
- name: first
  hostname: first.puppetdebug.vlan
  box: puppetlabs/centos-7.2-64-nocm
  roles:
  - pe-puppet-agent

So I went ahead and stopped firewalld which has gotten me past the connectivity issue it seems:

work/pe » vagrant ssh pe-puppet-master
Last login: Tue Jan  9 21:59:42 2018 from 10.0.2.2
[vagrant@master ~]$ sudo systemctl stop firewalld
[vagrant@master ~]$ logout
Connection to 127.0.0.1 closed.
work/pe » vagrant provision first
==> first: Running provisioner: hosts...
==> pe-puppet-master: Updating hosts on: pe-puppet-master
==> first: Running provisioner: pe_agent...
==> first: Running: set -e
==> first: curl -ksS --tlsv1 https://pe-puppet-master:8140/packages/current/install.bash -o pe_frictionless_installer.sh
==> first: bash pe_frictionless_installer.sh
    first: Running: inline script
    first: Loaded plugins: fastestmirror
    first: Cleaning repos: pe_repo
    first: Cleaning up everything
    first: Cleaning up list of fastest mirrors
    first: Loaded plugins: fastestmirror
    first: Error: No matching Packages to list
    first: Loaded plugins: fastestmirror

So how do I update the roles.yaml to shell provision systemctl stop firewalld or add the appropriate firewall rules (i.e. tcp/8140)

[Sharpie]

Yeah, the default firewall behavior in RedHat variants will block connections from agents to port 8140 on the master. The puppet-debugging-kit uses the following role to knock that out:

https://github.com/Sharpie/puppet-debugging-kit/blob/cd7909c766159045e129cb213469436bfdf077ab/data/puppet_debugging_kit/roles.yaml#L72-L86

As for pe-puppet-master, that was coming from the master: pe-puppet-master value in the pe-puppet-agent role. If you use a different name: for the master VM, then that setting can be updated to match.

References across YAML files are weird, so there isn't a great way around making an assumption there about the VM name of the master there :/