Merge pull request #181 from DavidS/hands-on-lab

(FM-8079) Resource API and Transports Hands-on-Lab

Author: DavidS

docs/README.md

@@ -0,0 +1,7 @@
+# Resource API hands-on lab
+
+The Resource API hands-on lab walks you through creating a native integration with Puppet. After completely this lab, you will have a fully functioning module to manage Philips HUE lights.
+
+>Note: These labs are intended for both new and experienced developers. If you have any feedback or suggestions for improvement, post it in the [issues section](https://github.com/puppetlabs/puppet-resource_api/issues).
+
+To start with, we'll go through [installing Puppet Development Kit](./hands-on-lab/01-installing-prereqs.md)(PDK).

docs/hands-on-lab/01-installing-prereqs.md

@@ -0,0 +1,16 @@
+# Install Puppet Development Kit (PDK) and other tools
+
+To start, install Puppet Development Kit (PDK), which provides all the necessary tools and libraries to build and test modules. We also recommend an emulator for the target device, a code editor with good Ruby and Puppet support, and git — a version control system to keep track of your progress.
+
+1. [Download PDK](https://puppet.com/download-puppet-development-kit) on your platform of choice.
+
+2. If you do not have a Philips HUE hub and bulbs available, you can download the [Hue-Emulator](https://github.com/SteveyO/Hue-Emulator/raw/master/HueEmulator-v0.8.jar). You need to have Java installed to run this.
+
+3. To edit code, we recommend the cross-platform editor [VSCode](https://code.visualstudio.com/download), with the [Ruby](https://marketplace.visualstudio.com/items?itemName=rebornix.Ruby) and [Puppet](https://marketplace.visualstudio.com/items?itemName=jpogran.puppet-vscode) extensions. There are lots of other extensions that can help you with your development workflow.
+
+4. Git is a version control system that helps you keep track of changes and collaborate with others. As we go through hands-on lab, we will show you some integrations with cloud services. If you have never used git before, ignore this and all related steps.
+
+
+## Next up
+
+After installing the relevant tools, you'll [light up a few](./02-connecting-to-the-lightbulbs.md).

docs/hands-on-lab/02-connecting-to-the-lightbulbs-emulator.png

@@ -0,0 +1,26 @@
+# Connecting to the light bulbs
+
+There are no technical restrictions on the kinds of remote devices or APIs you can connect to with transports. For this lab, we will connect to a Philips HUE hub and make some colourful wireless light bulbs light up. If you (understandably) do not have physical devices available, you can use the Hue Emulator.
+
+## Hue Emulator
+
+Use `java -jar` with the emulator's filename to run it, for example:
+
+```
+david@davids:~$ java -jar ~/Downloads/HueEmulator-v0.8.jar
+```
+
+It does not produce any output on the command line, but a window pops up with a hub and a few predefined lights:
+
+![](./02-connecting-to-the-lightbulbs-emulator.png)
+
+All you need now is to input a port (the default 8000 is usually fine) and click "Start" to activate the built-in server.
+
+## Connecting to your hub
+
+To connect to an actual hub, you need be able to access the bub on your network and get an API key. See the [Philips Developer docs](http://www.developers.meethue.com/documentation/getting-started) (registration required).
+
+
+## Next up
+
+Now that you have some lights up, you'll [create a module](./03-creating-a-new-module.md).

docs/hands-on-lab/02-connecting-to-the-lightbulbs.md

@@ -0,0 +1,47 @@
+# Create a module
+
+Depending on your preferences, you can use the VSCode/PDK integration or run PDK from the command line in a terminal of your choice.
+
+## Create a module with VSCode
+
+Spin up the Command Palette (⇧⌘P on the Mac or Ctrl+Shift+P on Windows and Linux) and search for the `Puppet: PDK New Module` task:
+
+![](./03-creating-a-new-module_vscode.png)
+
+Click Enter (↩) to execute this and follow the on-screen prompts.
+
+The module will open in a new VSCode window.
+
+## Create a module from the command line
+
+In your regular workspace (for example your home directory), run the following:
+
+```
+pdk new module hue_workshop --skip-interview
+```
+
+This command creates a new module `hue_workshop` in your directory of the same name, using all defaults. The output will look like:
+
+```
+david@davids:~/tmp$ pdk new module hue_workshop --skip-interview
+pdk (INFO): Creating new module: hue_workshop
+pdk (INFO): Module 'hue_workshop' generated at path '/home/david/tmp/hue_workshop', from template 'file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git'.
+pdk (INFO): In your module directory, add classes with the 'pdk new class' command.
+david@davids:~/tmp$ ls hue_workshop/
+appveyor.yml  data      files    Gemfile.lock  manifests      Rakefile   spec   templates
+CHANGELOG.md  examples  Gemfile  hiera.yaml    metadata.json  README.md  tasks
+david@davids:~/tmp$
+```
+
+To read more about the different options when creating new modules, see [PDK docs](https://puppet.com/docs/pdk/1.x/pdk_creating_modules.html).
+
+Open the new directory in your code editor:
+
+```
+code -a hue_workshop
+```
+
+
+## Next up
+
+Now that you have created a module, you'll [add a transport](./04-adding-a-new-transport.md).

docs/hands-on-lab/03-creating-a-new-module.md

@@ -0,0 +1,123 @@
+# Add a new transport
+
+Starting with PDK 1.12.0 there is the `pdk new transport` command, that you can use to create the base files for your new transport:
+
+Next, we'll active a few future defaults. In the `hue_workshop` directory, create a file called `.sync.yml` and paste the following:
+
+```
+# .sync.yml
+---
+Gemfile:
+  optional:
+    ':development':
+      - gem: 'puppet-resource_api'
+      - gem: 'faraday'
+      - gem: 'rspec-json_expectations'
+spec/spec_helper.rb:
+  mock_with: ':rspec'
+```
+
+Run `pdk update` in the module's directory to deploy the changes in the module:
+
+```
+david@davids:~/tmp/hue_workshop$ pdk update --force
+pdk (INFO): Updating david-hue_workshop using the default template, from 1.10.0 to 1.10.0
+
+----------Files to be modified----------
+Gemfile
+spec/spec_helper.rb
+
+----------------------------------------
+
+You can find a report of differences in update_report.txt.
+
+Do you want to continue and make these changes to your module? Yes
+
+------------Update completed------------
+
+2 files modified.
+
+david@davids:~/tmp/hue_workshop$
+```
+
+Then, create the actual transport:
+
+```
+david@davids:~/tmp/hue$ pdk new transport hue
+pdk (INFO): Creating '/home/david/tmp/hue/lib/puppet/transport/hue.rb' from template.
+pdk (INFO): Creating '/home/david/tmp/hue/lib/puppet/transport/schema/hue.rb' from template.
+pdk (INFO): Creating '/home/david/tmp/hue/lib/puppet/util/network_device/hue/device.rb' from template.
+pdk (INFO): Creating '/home/david/tmp/hue/spec/unit/puppet/transport/hue_spec.rb' from template.
+pdk (INFO): Creating '/home/david/tmp/hue/spec/unit/puppet/transport/schema/hue_spec.rb' from template.
+david@davids:~/tmp/hue$
+```
+
+## Checkpoint
+
+To validate your new module and transport, run `pdk validate --parallel` and `pdk test unit`:
+
+```
+david@davids:~/tmp/hue$ pdk validate --parallel
+pdk (INFO): Running all available validators...
+pdk (INFO): Using Ruby 2.5.5
+pdk (INFO): Using Puppet 6.4.2
+┌ [✔] Validating module using 5 threads ┌
+├──[✔] Checking metadata syntax (metadat├──son tasks/*.json).
+├──[✔] Checking task names (tasks/**/*).├──
+└──[✔] Checking YAML syntax (["**/*.yaml├──"*.yaml", "**/*.yml", "*.yml"]).
+└──[/] Checking module metadata style (metadata.json).
+└──[✔] Checking module metadata style (metadata.json).
+info: puppet-syntax: ./: Target does not contain any files to validate (**/*.pp).
+info: task-metadata-lint: ./: Target does not contain any files to validate (tasks/*.json).
+info: puppet-lint: ./: Target does not contain any files to validate (**/*.pp).
+david@davids:~/tmp/hue$ pdk test unit
+pdk (INFO): Using Ruby 2.5.5
+pdk (INFO): Using Puppet 6.4.2
+[✔] Preparing to run the unit tests.
+[✔] Running unit tests in parallel.
+Run options: exclude {:bolt=>true}
+  Evaluated 6 tests in 2.405066937 seconds: 0 failures, 0 pending.
+david@davids:~/tmp/hue$
+```
+
+If you're working with a version control system, now would be a good time to make your first commit and store the boilerplate code, and then you can revisit the changes you made later. For example:
+
+```
+david@davids:~/tmp/hue$ git init
+Initialized empty Git repository in ~/tmp/hue/.git/
+david@davids:~/tmp/hue$ git add -A
+david@davids:~/tmp/hue$ git commit -m 'initial commit'
+[master (root-commit) 67951dd] initial commit
+ 26 files changed, 887 insertions(+)
+ create mode 100644 .fixtures.yml
+ create mode 100644 .gitattributes
+ create mode 100644 .gitignore
+ create mode 100644 .gitlab-ci.yml
+ create mode 100644 .pdkignore
+ create mode 100644 .puppet-lint.rc
+ create mode 100644 .rspec
+ create mode 100644 .rubocop.yml
+ create mode 100644 .sync.yml
+ create mode 100644 .travis.yml
+ create mode 100644 .yardopts
+ create mode 100644 CHANGELOG.md
+ create mode 100644 Gemfile
+ create mode 100644 README.md
+ create mode 100644 Rakefile
+ create mode 100644 appveyor.yml
+ create mode 100644 data/common.yaml
+ create mode 100644 hiera.yaml
+ create mode 100644 lib/puppet/transport/hue.rb
+ create mode 100644 lib/puppet/transport/schema/hue.rb
+ create mode 100644 lib/puppet/util/network_device/hue/device.rb
+ create mode 100644 metadata.json
+ create mode 100644 spec/default_facts.yml
+ create mode 100644 spec/spec_helper.rb
+ create mode 100644 spec/unit/puppet/transport/hue_spec.rb
+ create mode 100644 spec/unit/puppet/transport/schema/hue_spec.rb
+david@davids:~/tmp/hue$
+```
+
+## Next up
+
+Now that you have everything ready, you'll [implement the transport](./05-implementing-the-transport.md).

docs/hands-on-lab/03-creating-a-new-module_vscode.png

@@ -0,0 +1,19 @@
+## Implementing the transport - Exercise
+
+Implement the `request_debug` option that you can toggle on to create additional debug output on each request. If you get stuck, review the hints below, or [the finished file](TODO).
+
+## Hints
+
+* You can create a toggle option with the `Boolean` (`true` or `false`) data type. Add it to the `connection_info` in the transport schema.
+
+* Make it an `Optional[Boolean]` so that users who do not require request debugging do not have to specify the value.
+
+* To remember the value you passed, store `connection_info[:request_debug]` in a `@request_debug` variable.
+
+* In the `hue_get` and `hue_put` methods, add `context.debug(message)` calls showing the method's arguments.
+
+* Make the debugging optional based on your input by appending `if @request_debug` to each logging statement.
+
+# Next Up
+
+Now that the transport can talk to the remote target, it's time to [implement a provider](./06-implementing-the-provider.md).

docs/hands-on-lab/04-adding-a-new-transport.md

@@ -0,0 +1,126 @@
+# Implementing the transport
+
+A transport consists of a *schema* describing the required data and credentials to connect to the HUE hub, and the *implementation* containing all the code to facilitate communication with the devices.
+
+## Schema
+
+The transport schema defines attributes in a reusable way, allowing you to understand the requirements of the transport. All schemas are located in `lib/puppet/transport/schema` in a Ruby file named after the transport. In this case `hue.rb`.
+
+To connect to the HUE hub you need an IP address, a port, and an API key.
+
+Replace the `connection_info` in `lib/puppet/transport/schema/hue.rb` with the following code:
+
+```ruby
+  connection_info: {
+    host: {
+      type: 'String',
+      desc: 'The FQDN or IP address of the hue light system to connect to.',
+    },
+    port: {
+      type: 'Optional[Integer]',
+      desc: 'The port to use when connecting, defaults to 80.',
+    },
+    key: {
+      type: 'String',
+      desc: 'The access key that allows access to the hue light system.',
+      sensitive: true,
+    },
+  },
+```
+
+> Note: The Resource API transports use [Puppet Data Types](https://puppet.com/docs/puppet/5.3/lang_data_type.html#core-data-types) to define the allowable values for an attribute. Abstract types like `Optional[]` can be useful to make using your transport easier. Take note of the `sensitive: true` annotation on the `key`; it instructs all services processing this attribute with special care, for example to avoid logging the key.
+
+
+## Implementation
+
+The implementation of a transport provides connectivity and utility functions for both Puppet and the providers managing the remote target. The HUE API is a simple REST interface, so you can store the credentials until you need make a connection. The default template at `lib/puppet/transport/hue.rb` already does this. Have a look at the `initialize` function to see how this is done.
+
+For the HUE's REST API, we want to create a `Faraday` object to capture the target host and key so that the transport can facilitate requests. Replace the `initialize` method in `lib/puppet/transport/hue.rb` with the following code:
+
+<!-- TODO: do we really need this? -- probably not ```
+    # @summary
+    #   Expose the `Faraday` object connected to the hub
+    attr_reader :connection
+
+```-->
+```
+    # @summary
+    #   Initializes and returns a faraday connection to the given host
+    def initialize(_context, connection_info)
+      # provide a default port
+      port = connection_info[:port].nil? ? 80 : connection_info[:port]
+      Puppet.debug "Connecting to #{connection_info[:host]}:#{port} with dev key"
+      @connection = Faraday.new(url: "http://#{connection_info[:host]}:#{port}/api/#{connection_info[:key].unwrap}", ssl: { verify: false })
+    end
+```
+
+> Note the `unwrap` call on building the URL, to access the sensitive value.
+
+### Facts
+
+The transport is also responsible for collecting any facts from the remote target, similar to how facter works for regular systems. For now we'll only return a hardcoded `operatingsystem` value to mark HUE Hubs:
+
+Replace the example `facts` method in `lib/puppet/transport/hue.rb` with the following code:
+
+```
+    # @summary
+    #   Returns set facts regarding the HUE Hub
+    def facts(_context)
+      { 'operatingsystem' => 'philips_hue' }
+    end
+```
+
+### Connection verification and closing
+
+To enable better feedback when something goes wrong, a transport can implement a `verify` method to run extra checks on the credentials passed in.
+
+To save resources both on the target and the node running the transport, the `close` method will be called when the transport is not needed anymore. The transport can close connections and release memory and other resources at this point.
+
+For this tutorial, replace the example methods with the following code:
+
+```
+    # @summary
+    #   Test that transport can talk to the remote target
+    def verify(_context)
+    end
+
+    # @summary
+    #   Close connection, free up resources
+    def close(_context)
+      @connection = nil
+    end
+```
+
+### Making requests
+
+Besides exposing some standardises functionality to Puppet, the transport is also a good place to put utility functions that can be reused across your providers. While it may seem overkill for this small example, it is no extra effort, and will establish a healthy pattern.
+
+Insert the following code after the `close` method:
+
+```
+    # @summary
+    #   Make a get request to the HUE Hub API
+    def hue_get(context, url, args = nil)
+      url = URI.escape(url) if url
+      result = @connection.get(url, args)
+      JSON.parse(result.body)
+    rescue JSON::ParserError => e
+      raise Puppet::ResourceError, "Unable to parse JSON response from HUE API: #{e}"
+    end
+
+    # @summary
+    #   Sends an update command to the given url/connection
+    def hue_put(context, url, message)
+      message = message.to_json
+      @connection.put(url, message)
+    end
+```
+
+## Exercise
+
+Implement a `request_debug` option that you can toggle to create additional debug output on each request. If you get stuck, have a look at [some hints](./05-implementing-the-transport-hints.md), or [the finished file](TODO).
+
+
+# Next Up
+
+Now that the transport can talk to the remote target, it's time to [implement a provider](./06-implementing-the-provider.md).