Milestones Development Discussion

[ju2wheels]

FYI, this represents my thoughts as of yesterday before the question of whether to move to fog-softlayer was raised in #30.

Goals:

• Enable bare metal server ordering

• Enable bare metal server instance ordering (different than above, its confusing but the order options are different and its important to distinguish the two since SL does but ordering is otherwise the same)

  • Distinguish Difference
  • To see the difference install the latest Python SL API and run:

  sl server list-chassis
  sl server create-options 32 #bare metal instance
  sl server create-options 50 #one type of bare metal server
  

• Enable advanced virtual server ordering (right now we only support the simplified ordering through createObject as opposed to generating a full Product Package order template so that we have ordering capabilities equal to what is available on the online ordering page). We should be able to support both transparently based on order options selected (or just go with advanced permanently and select defaults for the optional stuff).

• Create tools and commands to assist in the above

Features:

1. Forward SSH Keys through user_data for Windows boxes:

   • Our baseline is Powershell 2.0, so unfortunately sending it as JSON wont work as we need Powershell 3.0. There is support for CSV converting though so we can send that and base64 encode if we want to allow end user to send more data values. Whatever format we send I would still like to be able to let the end user add more data along with the ssh keys.
   • Limit to 2000 char according to this example found during my discussion with Scott.
   • Based on discussion with SL support, SoftLayer_Resource_Metadata service needed to retrieve user_data during post install is accessible over the private API endpoint without having to provide API credentials but no other service will be avialable (I havent actually verified that no other service is available).

2. post_install script framework development described in [new feature] Enable post_install framework and Windows WinRM/Cygwin provisioning #29 and discussed in Configuration of SL_WIN_LATEST_64 is a mystery #24:

   • Status:
     • Winrm: As seen through initial work to check the state of SL Windows images we have available, the Windows images we have dont appear to be as stock and unmodified as initially thought and the supported versions of Windows Management Framework and .NET Framework varies wildly by Windows releases. The goal of development here is to create a generalized Windows Management Framework and .NET "normalizer" script that gets every version of Windows up to the latest available compatible version of Windows Management Framework and .NET Framework and a separate script to configure WinRM (assuming a baseline of WinRM 2.0) with HTTP/HTTPS. Im hoping that by leveraging this helper cmdlet and these steps for creating a self signed cert that we can enable both HTTPS with AD cert (which is the default option built into WinRM) and self signed cert and be ahead of the curve for upstream's introduction of WinRM HTTPs support which is currently being worked on.
   • Issues/Blockers/Research:
     • Completion of the WinRm/.NET normalizer requires us to figure out how to get at 'Add/Remove Components' feature through Powershell in Windows versions older than Windows 2008 R2 which was replaced by Server Manager which has a nice cmdlet.
     • Need to finalize the matrix of what is available for install for each Windows release between 'Add/Remove Components', Server Manager, and WSUS/Windows update for WinRM/.Net and whether or not we will need to grab the installers from Microsoft Download to complete normalization.
     • post_install stability is insanely sketchy on Windows (dont know if its just me and my account). I can reliably get post_install hook on Windows 2003 images to fail to download the post_install script almost 100% of the time and sporadically on other versions. I have an open ticket to SL about this.
     • Self signed HTTPS WinRM support will require that we figure out how to pull the cert thumbprint out of registry using Powershell.

3. Cleanup CreateInstance in order to begin migration over to Softlayer API order abstractions (BareMetalServerOrder, BareMetalServerOrder_Package, VirtualServerOrder, and VirtualServerOrder_Package):

   • Based on discussion with Scott about adding VirtualServerOrder_Package, we can copy BareMetalServerOrder_Package and with a few changes have something that works. Not sure if they would take it as part of 2.x or 3.x if we send PR though, but last I tested we are compatible with both.
   • Cleanup would need to change CreateInstance#order_template to return lambda blocks specific to each of the order wrapper's verifyOrder/placeOrder calls instead of the template hash itself.
   • A lot of the churn in adding Bare Metal and Virtual Server advanced ordering is in CreateInstance. One of the issues I see is that most of the advanced ordering wants id's that represent the order selection (similar to the situation we had with vlans). If we choose to simplify this for the end user to allow id or description string then we would have to do the translation for them like we did with vlans but using configuration_options of each order wrapper.

4. Introduce options and changes for bare metal server and virtual server advanced ordering while still retaining backwards compatibility with old provider versions:

   • The basics here is that whatever we add we should still be able to use old provider configs without problems. In order to that, I propose the addition of sl.server_type (bare_metal, bare_metal_instance(may be merged with bare_metal), and virtual_server(default)) and if we support both simple/advanced order an sl.order_type (simple, or advanced) but only if we cant automatically determine that.

5. Add vagrant-softlayer-productpackage contrib tool to help end users figure out what acceptable values are when ordering the different servers similar to how we have vagrant-softlayer-vlans:

   • Based on Scott's ordering article advanced ordering follows these steps:
     1. Select the container for the product package you are ordering which represents the hash of options required to place an order for it (identified by SL API data types that start with SoftLayer_Container_Product_Order_*).
     2. Select the product package categories and optionally only those that are required.
     3. Select product category item codes and use them for purchasing.
   • vagrant-softlayer-productpackage will then effectively be an end user tool to help them answer what options do I have for filling in the provider options, proposed interface:

   #List packages (id and name) (for now will limit to virtual server, bare metal server and bare metal instance)
   #Packages will translate to those provided by SoftLayer::ProductPackage virtual_server_package, bare_metal_instance, and bare_metal_servers methods
   vagrant-softlayer-productpackage [--filter <name_regex>]
   #List package categories (id and name)
   vagrant-softlayer [--filter <name_regex>] [--required-only] [product_package id or name]
   #List package category items (id, name, and capacity/units)
   vagrant-softlayer [--filter <name_regex>] [product_package id or name] [product_package category id or name]
   

6. Add vagrant sl-vs-upgrade component amount [server] command for upgrading virtual server components using SoftLayer::VirtualServer upgrade order calls.

7. Refactor vagrant-softlayer-boxes once we finalize the provider format for bare metal/advanced ordering so it can handle create boxes for all types and fix bug Update boxes to set default user relevant to OS so end user doesnt have to explicitly override #25.

8. Move boxes section out of the quickstart guide and into the wiki, documenting the standards of what can be expected in the boxes. This list will probably be too large once we add support for bare metal.

Basically I just want to get to a point where we have feature parity with the online purchase pages and ensure that if we add advanced ordering functionality that the provider interface doesnt become overly complicated and unintuitive (such as requiring id's only where string description would make the Vagrantfile more readable).

Once we finish the discussion on fog-softlayer id like to break all of these out into separate issues and start discussing how to handle 3/4 which I think are the most complicated.

[ju2wheels]

0.5.0 dev is up in develop with the new vagrant-softlayer-credentials tool for credential (password) management. If anyone is interested in testing theres a few areas that could use testing:

1. Application Delivery Controller (SL load balancers i think) functionality:

   I dont have any of these attached to my account so implemented it blind and im not sure if the filters are correct. I would be interested to know if you have any application delivery controllers in your account if its accurately listing the credential users. If it doesnt it would help if someone can provide whether the advancedModeFlag of their controllers returns a value of true/false or 0/1 for the following query:

   require 'rubygems'
   require 'softlayer_api'
   sl_client = SoftLayer::Client.new(:api_key => ENV["SL_API_KEY"], :username => ENV["SL_API_USERNAME"], :timeout => 240)
   pp sl_client[:Account].object_mask("mask[advancedModeFlag]").getApplicationDeliveryControllers.map{|adc| adc['advancedModeFlag'] }
   

2. Network message delivery, network storage, and network vlan firewall set commands remain untested as I dont have any non production stuff to test it on.

Im more interested in how the network storage password setting works out as the documentation on which network storage types actually support having their password set is very confusing so I have no idea which storage types will actually work. I was able to get it to list the credentials across all my existing storage though.

Theres no support for webcc credential password changing at the moment.

Other than this i hope to get the vagrant-softlayer-products interface tool into this release with the advanced virtual guest ordering functionality.