Merge pull request #92 from puppetlabs/add-google-clouddns

(RE-15124) Bump vmpooler, providers, and add new dns gem

Author: yachub

.gitignore

@@ -1,5 +1,6 @@
 /*/**/vendor/bundle/
 /docker/**/vmpooler.yaml
+/docker/data/*.json
 .vagrant/
 .idea/
 helm-charts/vmpooler/charts/

README.md

@@ -1,16 +1,15 @@
 # vmpooler-deployment
 
-- [vmpooler-deployment](#vmpooler-deployment)
-  - [VMPooler Components](#vmpooler-components)
-  - [Docker Registry](#docker-registry)
-  - [Helm Repository](#helm-repository)
-    - [Adding / updating charts](#adding--updating-charts)
-  - [Development](#development)
-    - [Docker Compose URLs](#docker-compose-urls)
-    - [Deploy Chart for Testing](#deploy-chart-for-testing)
-  - [Releasing](#releasing)
-  - [Contributing](#contributing)
-  - [License](#license)
+- [VMPooler Components](#vmpooler-components)
+- [Docker Registry](#docker-registry)
+- [Helm Repository](#helm-repository)
+  - [Adding / updating charts](#adding--updating-charts)
+- [Development](#development)
+  - [Docker Compose URLs](#docker-compose-urls)
+  - [Deploy Chart for Testing](#deploy-chart-for-testing)
+- [Releasing](#releasing)
+- [Contributing](#contributing)
+- [License](#license)
 
 This repo contains Dockerfiles and a Helm chart that can be used to deploy [VMPooler](https://github.com/puppetlabs/vmpooler). The Release Engineering team at Puppet uses the code here as part of operating our VMPooler instances.
 
@@ -19,6 +18,7 @@ This repo contains Dockerfiles and a Helm chart that can be used to deploy [VMPo
 The docker image gnerated and hosted by this project contain the following VMPooler components:
 
 - [VMPooler Core](https://github.com/puppetlabs/vmpooler)
+- [VMPooler Google CloudDNS Plugin](https://github.com/puppetlabs/vmpooler-dns-google-clouddns)
 - [VMPooler EC2 Provider](https://github.com/puppetlabs/vmpooler-provider-ec2)
 - [VMPooler GCE Provider](https://github.com/puppetlabs/vmpooler-provider-gce)
 - [VMPooler vSphere Provider](https://github.com/puppetlabs/vmpooler-provider-vsphere)
@@ -56,13 +56,16 @@ Prerequisites:
       - [vmpooler-provider-ec2](https://github.com/puppetlabs/vmpooler-provider-ec2)
       - [vmpooler-provider-gce](https://github.com/puppetlabs/vmpooler-provider-gce)
       - [vmpooler-provider-vsphere](https://github.com/puppetlabs/vmpooler-provider-vsphere)
+   - If you are not using Dynamic DNS, then the following DNS plugins can be used to manage records across different compute providers:
+     - [VMPooler Google CloudDNS Plugin](https://github.com/puppetlabs/vmpooler-dns-google-clouddns)
 2. Chose a Development method:
    - Develop via local source:
       1. Clone all of the known vmpooler projects listed at [vmpooler-components](#vmpooler-components) under a common directory, for example:
 
            ```bash
            |-- vmpooler-projects
            |  |--vmpooler-deployment
+           |  |--vmpooler-dns-google-clouddns
            |  |--vmpooler-provider-ec2
            |  |--vmpooler-provider-gce
            |  |--vmpooler-provider-vsphere
@@ -72,10 +75,10 @@ Prerequisites:
       3. Run `docker compose -f vmpooler-deployment/docker/docker-compose.local.yml up`
 
    - Develop via Git source:
-      1. For the component(s) you are developing on, commit and push changes to a branch.
-      2. Change to the `docker` directory and modify the `Gemfile` as needed to pull the gem(s) from your git branch.
-      3. Run `./update-gemfile-lock` to update the `Gemfile.lock`
-      4. Run `docker compose build && docker compose up`.
+      4. For the component(s) you are developing on, commit and push changes to a branch.
+      5. Change to the `docker` directory and modify the `Gemfile` as needed to pull the gem(s) from your git branch.
+      6. Run `./update-gemfile-lock` to update the `Gemfile.lock`
+      7. Run `docker compose build && docker compose up`.
 
 When a dependency Helm chart is updated, be sure to run `./update-chart-lock` to update the lockfile, otherwise the test and release workflows will fail.
 

docker/Dockerfile.local

@@ -25,6 +25,7 @@ RUN apt-get update -qq && \
     rm -rf /var/lib/apt/lists/*
 
 COPY vmpooler /opt/vmpooler
+COPY vmpooler-dns-gcp /opt/vmpooler-dns-gcp
 COPY vmpooler-provider-ec2 /opt/vmpooler-provider-ec2
 COPY vmpooler-provider-gce /opt/vmpooler-provider-gce
 COPY vmpooler-provider-vsphere /opt/vmpooler-provider-vsphere

docker/Gemfile

@@ -1,13 +1,15 @@
 source ENV['GEM_SOURCE'] || 'https://rubygems.org'
 
-gem 'vmpooler',                    '~> 2.5'
+gem 'vmpooler',                    '~> 3.0'
+gem 'vmpooler-dns-gcp',            '~> 1.0'
 # Remove temporarily until a stable minor release is published
 # gem 'vmpooler-provider-ec2',       '~> 0.0.1'
-gem 'vmpooler-provider-gce',       '~> 0.5'
-gem 'vmpooler-provider-vsphere',   '~> 2.1'
+gem 'vmpooler-provider-gce',       '~> 1.0'
+gem 'vmpooler-provider-vsphere',   '~> 3.0'
 
 # For development install via a git branch use something like:
 # gem 'vmpooler', git: 'https://github.com/puppetlabs/vmpooler.git', branch: 'my-feature'
+# gem 'vmpooler-dns-gcp', git: 'https://github.com/puppetlabs/vmpooler-dns-gcp.git', branch: 'my-feature'
 # gem 'vmpooler-provider-ec2', git: 'https://github.com/puppetlabs/vmpooler-provider-ec2.git', branch: 'my-feature'
 # gem 'vmpooler-provider-gce', git: 'https://github.com/puppetlabs/vmpooler-provider-gce.git', branch: 'my-feature'
 # gem 'vmpooler-provider-vsphere', git: 'https://github.com/puppetlabs/vmpooler-provider-vsphere.git', branch: 'my-feature'

docker/Gemfile.local

@@ -1,6 +1,7 @@
 source ENV['GEM_SOURCE'] || 'https://rubygems.org'
 
 gem 'vmpooler', path: '/opt/vmpooler'
+gem 'vmpooler-dns-gcp', path: '/opt/vmpooler-dns-gcp'
 gem 'vmpooler-provider-ec2', path: '/opt/vmpooler-provider-ec2'
 gem 'vmpooler-provider-gce', path: '/opt/vmpooler-provider-gce'
 gem 'vmpooler-provider-vsphere', path: '/opt/vmpooler-provider-vsphere'

docker/Gemfile.lock

@@ -1,19 +1,19 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    addressable (2.8.1)
+    addressable (2.8.4)
       public_suffix (>= 2.0.2, < 6.0)
     bindata (2.4.15)
     builder (3.2.4)
     concurrent-ruby (1.2.2)
-    connection_pool (2.3.0)
+    connection_pool (2.4.0)
     declarative (0.0.20)
     deep_merge (1.2.2)
     faraday (2.7.4)
       faraday-net_http (>= 2.0, < 3.1)
       ruby2_keywords (>= 0.0.4)
     faraday-net_http (3.0.2)
-    google-apis-compute_v1 (0.63.0)
+    google-apis-compute_v1 (0.66.0)
       google-apis-core (>= 0.11.0, < 2.a)
     google-apis-core (0.11.0)
       addressable (~> 2.5, >= 2.5.1)
@@ -24,7 +24,7 @@ GEM
       retriable (>= 2.0, < 4.a)
       rexml
       webrick
-    google-apis-dns_v1 (0.31.0)
+    google-apis-dns_v1 (0.32.0)
       google-apis-core (>= 0.11.0, < 2.a)
     google-cloud-core (1.6.0)
       google-cloud-env (~> 1.0)
@@ -52,9 +52,9 @@ GEM
     multi_json (1.15.0)
     mustermann (2.0.2)
       ruby2_keywords (~> 0.0.1)
-    net-ldap (0.17.1)
-    nio4r (2.5.8-java)
-    nokogiri (1.14.2-java)
+    net-ldap (0.18.0)
+    nio4r (2.5.9-java)
+    nokogiri (1.14.3-java)
       racc (~> 1.4)
     opentelemetry-api (1.1.0)
     opentelemetry-common (0.19.6)
@@ -101,7 +101,7 @@ GEM
     puma (5.6.5-java)
       nio4r (~> 2.0)
     racc (1.6.2-java)
-    rack (2.2.6.3)
+    rack (2.2.6.4)
     rack-protection (2.2.4)
       rack
     rake (13.0.6)
@@ -135,7 +135,7 @@ GEM
     tilt (2.1.0)
     trailblazer-option (0.1.2)
     uber (0.1.0)
-    vmpooler (2.5.0)
+    vmpooler (3.0.0)
       concurrent-ruby (~> 1.1)
       connection_pool (~> 2.2)
       deep_merge (~> 1.2)
@@ -156,24 +156,28 @@ GEM
       sinatra (~> 2.0)
       spicy-proton (~> 2.1)
       statsd-ruby (~> 1.4)
-    vmpooler-provider-gce (0.5.0)
-      google-apis-compute_v1 (~> 0.14)
+    vmpooler-dns-gcp (1.0.0)
       google-cloud-dns (~> 0.35.1)
       googleauth (>= 0.16.2, < 1.3.0)
-      vmpooler (~> 2.3, >= 1.3.0)
-    vmpooler-provider-vsphere (2.1.0)
+      vmpooler (~> 3.0)
+    vmpooler-provider-gce (1.0.0)
+      google-apis-compute_v1 (~> 0.14)
+      googleauth (>= 0.16.2, < 1.3.0)
+      vmpooler (~> 3.0)
+    vmpooler-provider-vsphere (3.0.0)
       rbvmomi2 (>= 3.1, < 4.0)
-      vmpooler (~> 2.4)
+      vmpooler (~> 3.0)
     webrick (1.8.1)
     zonefile (1.06)
 
 PLATFORMS
   universal-java-11
 
 DEPENDENCIES
-  vmpooler (~> 2.5)
-  vmpooler-provider-gce (~> 0.5)
-  vmpooler-provider-vsphere (~> 2.1)
+  vmpooler (~> 3.0)
+  vmpooler-dns-gcp (~> 1.0)
+  vmpooler-provider-gce (~> 1.0)
+  vmpooler-provider-vsphere (~> 3.0)
 
 BUNDLED WITH
-   2.4.7
+   2.4.8

docker/data/vmpooler.yaml

@@ -1,5 +1,18 @@
 ---
+:dns_configs:
+  ddns_domain: 'dynamic.example.com'
+  :gcp-clouddns:
+    project: vmpooler-example
+    domain: vmpooler.example.com
+    dns_zone_resource_name: vmpooler-example-com
+
 :providers:
+  # :gce:
+  #   network_name: projects/vmpooler-example/global/networks/default
+  #   project: vmpooler-example
+  #   title: gce
+  #   zone: us-west1-b
+
   :dummy:
     filename: '/tmp/dummy-backing.yaml'
 
@@ -23,13 +36,19 @@
   allowed_tags:
     - 'created_by'
     - 'project'
-  domain: 'example.com'
-  # domain: 'localhost' # Flip these out for local requests
 
 :pools:
-  - name: 'debian-10-x86_64'
-    template: 'Templates/debian-7-x86_64'
-    folder: 'Pooled VMs/debian-7-x86_64'
+  # - name: 'fedora-36-x86_64'
+  #   template: 'projects/fedora-cloud/global/images/fedora-cloud-base-gcp-36-20220506-n-0-x86-64'
+  #   size: 0
+  #   provider: 'gce'
+  #   dns_plugin: 'google-clouddns'
+  #   machine_type: 'zones/us-west1-b/machineTypes/n1-standard-2'
+  #   disk_type: 'pd-ssd'
+  #   subnetwork_name: 'projects/vmpooler-example/regions/us-west1/subnetworks/default'
+  - name: 'debian-11-x86_64'
+    template: 'Templates/debian-11-x86_64'
+    folder: 'Pooled VMs/debian-11-x86_64'
     datastore: 'vmstorage'
     size: 5
     timeout: 15

docker/docker-compose.local.yml

@@ -41,16 +41,21 @@ services:
       - type: bind
         source: ./data/vmpooler.yaml
         target: /etc/vmpooler/vmpooler.yaml
+      # - type: bind
+      #   source: ./data/vmpooler-example-key.json
+      #   target: /etc/vmpooler/vmpooler-example-key.json
     ports:
       - "8082:4567"
     environment:
       - VMPOOLER_DEBUG=true # for use of dummy auth
+      - DEBUG_FLAG=true
       - VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml
       - REDIS_SERVER=redis-server
       - REDIS_PASSWORD=vmpooler
       - LOGFILE=/dev/null
       - VMPOOLER_TRACING_ENABLED=true
       - VMPOOLER_TRACING_JAEGER_HOST=http://jaeger-aio:14268/api/traces
+      # - GOOGLE_APPLICATION_CREDENTIALS=/etc/vmpooler/vmpooler-example-key.json
     command: api
     links:
       - redis-server
@@ -63,16 +68,21 @@ services:
       - type: bind
         source: ./data/vmpooler.yaml
         target: /etc/vmpooler/vmpooler.yaml
+      # - type: bind
+      #   source: ./data/vmpooler-example-key.json
+      #   target: /etc/vmpooler/vmpooler-example-key.json
     ports:
       - "8083:4567"
     environment:
       - VMPOOLER_DEBUG=true # for use of dummy auth
+      - DEBUG_FLAG=true
       - VMPOOLER_CONFIG_FILE=/etc/vmpooler/vmpooler.yaml
       - REDIS_SERVER=redis-server
       - REDIS_PASSWORD=vmpooler
       - LOGFILE=/dev/null
       - VMPOOLER_TRACING_ENABLED=true
       - VMPOOLER_TRACING_JAEGER_HOST=http://jaeger-aio:14268/api/traces
+      # - GOOGLE_APPLICATION_CREDENTIALS=/etc/vmpooler/vmpooler-example-key.json
     command: manager
     links:
       - redis-server

helm-charts/vmpooler/templates/deployment-api.yaml

@@ -46,13 +46,13 @@ spec:
           - 'api'
         livenessProbe:
           httpGet:
-            path: /api/v1/status
+            path: /api/v3/status
             port: {{ .Values.service.port }}
           initialDelaySeconds: 40
           periodSeconds: 10
         readinessProbe:
           httpGet:
-            path: /api/v1/status
+            path: /api/v3/status
             port: {{ .Values.service.port }}
           initialDelaySeconds: 40
           periodSeconds: 10
@@ -63,8 +63,6 @@ spec:
           value: {{ .Values.service.port | quote  }}
         - name: SITE_NAME
           value: '<b>{{ .Values.vmpoolerInstance }}</b>.{{ .Values.ingress.domain }}'
-        - name: DOMAIN
-          value: {{ .Values.sut_domain }}
         - name: EXPERIMENTAL_FEATURES
           value: 'true'
         - name: USAGE_STATS

helm-charts/vmpooler/templates/deployment-manager.yaml

@@ -93,8 +93,6 @@ spec:
           value: '20'
         - name: PURGE_UNCONFIGURED_FOLDERS
           value: 'true'
-        - name: DOMAIN
-          value: {{ .Values.sut_domain }}
         - name: VMPOOLER_CONFIG_FILE
           value: /etc/vmpooler/vmpooler.yaml
         - name: VMPOOLER_TRACING_ENABLED