Switched connection to include port and to use built in module to format URL instead of doing it with interpolation. Added documentation to readme about how to connect to different databases with examples

Author: bishopbm2

README.md

@@ -28,10 +28,12 @@ yum install unixODBC-devel
 2. Execute an action (example: query)
 
     ``` shell
-    st2 run sql.query host=test_serve.domain.tld username=test_user password=test_password database=test_database database_type=postgresql query="select * from test;"
+    st2 run sql.query host=test_serve.domain.tld username=test_user password=test_password database=test_database drivername=postgresql query="select * from test;"
     ```
 
-## Configuration
+## Configuration and Connecting to Databases
+Connecting to different types of databases is shown below. Connecting to different databases is done in the same manor except with sqlite where all you need to pass is the path to the database in the database option. This is show below. For more information about connections please refer to [SQLAlchemy Connection Docs](https://docs.sqlalchemy.org/en/latest/core/engines.html)
+
 Copy the example configuration in [sql.yaml.example](./sql.yaml.example)
 to `/opt/stackstorm/configs/sql.yaml` and edit as required.
 
@@ -45,13 +47,17 @@ sql:
     username: [email protected]
     password: Password
     database: TestDatabase
-    database_type: postgresql
+    port: 5432
+    drivername: postgresql
   mysql:
     host: mysql_db.domain.tld
     username: [email protected]
     password: NewPassword
     database: TestDatabase
-    database_type: mysql
+    drivername: mysql
+  sqlite:
+    database: /path/to/db.sqlite
+    drivername: sqlite
 ```
 
 Each entry should contain
@@ -60,12 +66,13 @@ Each entry should contain
 * ``username`` - Username to authenticate to DB
 * ``password`` - Password for DB authentication
 * ``database`` - Database to use
-* ``database_type`` - The type of database that is being connected to.
+* ``port`` - Port to connect to database on. If Default leave blank
+* ``drivername`` - The type of database that is being connected to.
 
 When running actions, you can pass in the name of a connection, e.g.
 `st2 run sql.query connection="postgresql" query="select * from test;"`
 
-Alternatively, when running an action, you can pass in the host, username, password, database, database_type parameters. These parameters can also be used for overrides if you wish to use the configs as well.
+Alternatively, when running an action, you can pass in the host, username, password, database, port, drivername parameters. These parameters can also be used for overrides if you wish to use the configs as well.
 
 **Note** : When modifying the configuration in `/opt/stackstorm/configs/` please remember to tell StackStorm to load these new values by running `st2ctl reload --register-configs`
 

actions/delete.yaml

@@ -27,7 +27,11 @@
       type: string
       description: "Optional override of the database in <connection> (required if <connection> is not specified). Database to connect to, to run querys against."
       required: false
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
       enum:

actions/insert.yaml

@@ -27,7 +27,11 @@
       type: string
       description: "Optional override of the database in <connection> (required if <connection> is not specified). Database to connect to, to run querys against."
       required: false
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
       enum:

actions/insert_bulk.yaml

@@ -27,7 +27,11 @@
       type: string
       description: "Optional override of the database in <connection> (required if <connection> is not specified). Database to connect to, to run querys against."
       required: false
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
       enum:

actions/lib/base_action.py

@@ -1,14 +1,16 @@
 from st2common.runners.base_action import Action
 import sqlalchemy
+from sqlalchemy.engine.url import URL
 import decimal
 import datetime
 
 #                         (key, required, default)
-CONFIG_CONNECTION_KEYS = [('host', True, ""),
-                          ('username', True, ""),
-                          ('password', True, ""),
+CONFIG_CONNECTION_KEYS = [('host', False, ""),
+                          ('username', False, ""),
+                          ('password', False, ""),
                           ('database', True, ""),
-                          ('database_type', True, "")]
+                          ('port', False, None),
+                          ('drivername', True, "")]
 
 
 class BaseAction(Action):
@@ -34,6 +36,9 @@ def get_del_arg(self, key, kwargs_dict, delete=False):
             return None
 
     def merge_dicts(self, dicts):
+        """ Merge Array of dictionaries into 1 single Dictionary
+        and return.
+        """
         result = {}
         for d in dicts:
             if d:
@@ -60,6 +65,11 @@ def row_to_dict(self, row):
         return return_dict
 
     def generate_where_clause(self, sql_table, sql_obj, where_dict):
+        """Generate WHERE Clause for sql queries with the proper
+        SQLAlchemy syntax and binding to the proper parameters with
+        overrides.
+        Returns Query object
+        """
         for key in where_dict.keys():
             # All column names are reserved. Adding a '_' to the begging of the name
             new_key = "_" + key
@@ -73,26 +83,30 @@ def generate_where_clause(self, sql_table, sql_obj, where_dict):
         return (sql_obj, where_dict)
 
     def generate_values(self, sql_obj, data_dict):
+        """Generate values statement for sql queries with the proper
+        SQLAlchemy syntax and binding to the proper parameters with
+        overrides.
+        Returns Query object
+        """
         for key in data_dict.keys():
             key_dictionary = {key: sqlalchemy.sql.bindparam(key)}
             sql_obj = sql_obj.values(**key_dictionary)
 
         return sql_obj
 
     def connect_to_db(self, connection):
-        database_connection_string = "{0}://{1}:{2}@{3}/{4}".format(connection['database_type'],
-                                                                    connection['username'],
-                                                                    connection['password'],
-                                                                    connection['host'],
-                                                                    connection['database'])
+        """Connect to the database and instantiate necessary methods to be used
+        later.
+        """
+        database_connection_string = URL(**connection)
 
         self.engine = sqlalchemy.create_engine(database_connection_string, echo=False)
         self.conn = self.engine.connect()
         self.meta = sqlalchemy.MetaData()
 
         return True
 
-    def validate_connection(self, connection):
+    def validate_connection(self, connection, connection_name):
         """Ensures that all required parameters are in the connection. If a
         required parameter is missing a KeyError exception is raised.
         :param connection: connection to validate
@@ -107,9 +121,9 @@ def validate_connection(self, connection):
             if not required:
                 continue
 
-            if connection['connection']:
+            if connection_name:
                 raise KeyError("config.yaml mising: sql:{0}:{1}"
-                               .format(connection['connection'], key))
+                               .format(connection_name, key))
             else:
                 raise KeyError("Because the 'connection' action parameter was"
                                " not specified, the following action parameter"
@@ -132,7 +146,7 @@ def resolve_connection(self, kwargs_dict):
                 raise KeyError("config.yaml missing connection: sql:{0}"
                                .format(connection_name))
 
-        action_connection = {'connection': connection_name}
+        action_connection = {}
 
         # Override the keys in creds read in from the config given the
         # override parameters from the action itself
@@ -155,6 +169,6 @@ def resolve_connection(self, kwargs_dict):
             if key in kwargs_dict:
                 del kwargs_dict[key]
 
-        self.validate_connection(action_connection)
+        self.validate_connection(action_connection, connection_name)
 
         return action_connection

actions/query.yaml

@@ -27,7 +27,11 @@
       type: string
       description: "Optional override of the database in <connection> (required if <connection> is not specified). Database to connect to, to run querys against."
       required: false
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
       enum:

actions/update.yaml

@@ -27,7 +27,11 @@
       type: string
       description: "Optional override of the database in <connection> (required if <connection> is not specified). Database to connect to, to run querys against."
       required: false
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
       enum:

config.schema.yaml

@@ -12,23 +12,27 @@ connection:
   type: object
   properties:
     host:
-      description: "Database server to connect to. If not using a default port add that here. ex. host.domain.tld or host.domain.tld:1234"
+      description: "Database server to connect to. ex. host.domain.tld"
       type: string
-      required: true
+      required: false
     username:
       description: "Username for authentication"
       type: string
-      required: true
+      required: false
     password:
       description: "Password of the specified username"
       type: string
       secret: true
-      required: true
+      required: false
     database:
       description: "Database to connect to, to run querys against."
       type: string
       required: true
-    database_type:
+    port:
+      description: "Port to connect to database on. If Default leave blank"
+      type: integer
+      required: false
+    drivername:
       description: "The type of database that is being connected to."
       type: string
       enum:

sql.yaml.example

@@ -5,10 +5,14 @@ sql:
     username: [email protected]
     password: Password
     database: TestDatabase
-    database_type: postgresql
+    port: 5432
+    drivername: postgresql
   mysql:
     host: mysql_db.domain.tld
     username: [email protected]
     password: NewPassword
     database: TestDatabase
-    database_type: mysql
+    drivername: mysql
+  sqlite:
+    database: /path/to/db.sqlite
+    drivername: sqlite