Merge pull request #3 from EncoreTechnologies/update/use_builtin_connectors

Use built in packages with overrides

Author: nmaludy

README.md

@@ -1,20 +1,7 @@
-# SQL Integration Pack
-Query, Insert, Update, and Delete information from PostgreSQL, SQLite, MsSQL, MySQL, Oracle, Firebird, and Sybase Databases
-
-## Pre-Requisites
-This pack is set up to provide funcationality for the above databases. For MySQL and MsSQL we need to install 2 system packages.
-
-#### [MySQL](https://pypi.org/project/mysqlclient/)
-``` shell
-yum install mysql-devel
-```
+[![Build Status](https://circleci.com/gh/StackStorm-Exchange/stackstorm-sql.svg?style=shield&circle-token=:circle-token)](https://circleci.com/gh/StackStorm-Exchange/stackstorm-sql) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 
-#### [MsSQL](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017)
-``` shell
-curl https://packages.microsoft.com/config/rhel/7/prod.repo > /etc/yum.repos.d/mssql-release.repo
-yum install msodbcsql17
-yum install unixODBC-devel
-```
+# SQL Integration Pack
+Query, Insert, Update, and Delete information from PostgreSQL, MsSQL, MySQL, Oracle, Firebird Databases. Additional databases can be added by installing the pre-requisite packes and passing the driver name as documented at [SQLAlchemy Dialects Doc](https://docs.sqlalchemy.org/en/latest/dialects/index.html).
 
 ## Quick Start
 
@@ -30,6 +17,17 @@ yum install unixODBC-devel
     st2 run sql.query host=test_serve.domain.tld username=test_user password=test_password database=test_database drivername=postgresql query="select * from test;"
     ```
 
+## Included Drivers
+This pack is already set up to connect to the databases listed above. Additional databases can be connected to but pre-requisite packages will need to be installed before the drivers will work. [SQLAlchemy Dialects Doc](https://docs.sqlalchemy.org/en/latest/dialects/index.html).
+
+You can pass any of the following driver names without any additional packages needing installed.
+* `postgresql` - PostgreSQL databases
+* `mssql` - Microsoft SQL Server databases
+* `mysql` - MySQL/MariaDB databases
+* `oracle` - Oracle databases
+* `firebird` - Firebird databases
+
+
 ## Configuration and Connecting to Databases
 Connecting to different types of databases is shown below. Connecting to different databases is done in the same manor except with sqlite where all you need to pass is the path to the database in the database option. This is show below. For more information about connections please refer to [SQLAlchemy Connection Docs](https://docs.sqlalchemy.org/en/latest/core/engines.html)
 
@@ -59,17 +57,20 @@ connections:
     drivername: sqlite
 ```
 
-Each entry should contain
+Each entry should contain:
 
-* ``host`` - Database hostname
-* ``username`` - Username to authenticate to DB
-* ``password`` - Password for DB authentication
-* ``database`` - Database to use
-* ``port`` - Port to connect to database on. If Default leave blank
-* ``drivername`` - The type of database that is being connected to.
+* `host` - Database hostname
+* `username` - Username to authenticate to DB
+* `password` - Password for DB authentication
+* `database` - Database to use
+* `port` - Port to connect to database on. If Default leave blank
+* `drivername` - The type of database that is being connected to.
 
 When running actions, you can pass in the name of a connection, e.g.
-`st2 run sql.query connection="postgresql" query="select * from test;"`
+
+``` shell
+st2 run sql.query connection="postgresql" query="SELECT * FROM test;"
+```
 
 Alternatively, when running an action, you can pass in the host, username, password, database, port, drivername parameters. These parameters can also be used for overrides if you wish to use the configs as well.
 
@@ -87,19 +88,143 @@ Alternatively, when running an action, you can pass in the host, username, passw
 | update | Update data in a database table. |
 | delete | Delete data from a database table. |
 
+### Action Example - sql.query
+
+`sql.query` can run any SQL query against a database. This can be used for simple `SELECT` 
+statements:
+
+```shell
+st2 run sql.query connection="postgresql" query="SELECT * FROM test;"
+```
+
+Workflow usage:
+
+``` yaml
+insert_data:
+  action: sql.query
+  input:
+    connection: postgresql
+    query: "SELECT * FROM test;"
+```
+
+This action is also the one to use if you have a complex SQL statement that you want to run,
+but there isn't another action in this pack that supports what you're trying to do.
+In this case, simply pass in your arbitrary SQL statement into the `query` parameter and
+it will be executed:
+
+```shell
+st2 run sql.query connection="postgresql" query="SELECT * FROM test JOIN somecrazytable ON id;"
+```
+
+### Action Example - sql.insert
+
+`sql.insert` is used to insert a single record into a table:
+
+```shell
+st2 run sql.insert connection="postgresql" table="people" data='{"name": "bob", "phone": "1234567890"}'
+```
+
+Workflow usage:
+
+``` yaml
+insert_data:
+  action: sql.insert
+  input:
+    connection: postgresql
+    table: "people"
+    data:
+      name: "bob"
+      phone: "1234567890"
+```
+
+### Action Example - sql.insert_bulk
+
+`sql.insert_bulk` is used to insert multiple records into a table. In this case the `data` 
+parameter expects an array of objects, where each object is a record to insert.
+
+```shell
+st2 run sql.insert connection="postgresql" table="people" data='[{"name": "bob", "phone": "1234567890"}, {"name": "alice", "phone": "0987654321"}]'
+```
+
+Workflow usage:
+
+``` yaml
+bulk_insert_data:
+  action: sql.insert_bulk
+  input:
+    connection: postgresql
+    table: "people"
+    data:
+      - name: "bob"
+        phone: "1234567890"
+      - name: "alice"
+        phone: "0987654321"
+```
+
+### Action Example - sql.update
+
+`sql.update` is used to update records in a table using simple `WHERE` clauses. 
+If you need to run complex `WHERE` conditions, then use the `sql.query` action instead.
+
+```shell
+st2 run sql.insert connection="postgresql" table="people" where='{"name": "bob"}' update='{"phone": "5551234"}'
+```
+
+Workflow usage:
+
+``` yaml
+update_data:
+  action: sql.update
+  input:
+    connection: postgresql
+    table: "people"
+    where:
+      name: "bob"
+    update:
+      phone: "5551234"
+```
+
+### Action Example - sql.delete
+
+`sql.delete` is used to delete records in a table using simple `WHERE` clauses. 
+If you need to run complex `WHERE` conditions, then use the `sql.query` action instead.
+
+```shell
+st2 run sql.insert connection="postgresql" table="people" where='{"name": "bob"}'
+```
+
+Workflow usage:
+
+``` yaml
+update_data:
+  action: sql.update
+  input:
+    connection: postgresql
+    table: "people"
+    where:
+      name: "bob"
+```
+
 ## Where statements
 
 The Update and Delete actions give the option to include where data into the query. This only works for AND statements.
 
-Example:
+Example (YAML for workflows):
+```yaml
+where:
+  column_1: "value_1"
+  column_2: "value_2"
 ```
-where = {
-    'column_1': 'value_1',
-    'column_2': 'value_2'
-}
 
-Produces the statement:
+Example (JSON for `st2` CLI):
+```shell
+where='{"column_1": "value_1", "column_2": "value_2"}'
+```
+
+Produces the SQL `WHERE` statement:
+
+```sql
 WHERE column_1 == 'value_1' AND column_2 == 'value_2'
 ```
 
-For more complicated queries please use the query action.
+For more complicated queries please use the `sql.query` action.

actions/delete.yaml

@@ -34,14 +34,6 @@
     drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: false
     table:
       type: string

actions/insert.yaml

@@ -34,14 +34,6 @@
     drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: false
     table:
       type: string

actions/insert_bulk.yaml

@@ -34,14 +34,6 @@
     drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: false
     table:
       type: string

actions/lib/base_action.py

@@ -13,6 +13,14 @@
                           ('port', False, None),
                           ('drivername', True, "")]
 
+DEFAULT_KNOWN_DRIVER_CONNECTORS = {
+    'postgresql': 'postgresql+psycopg2',
+    'mysql': 'mysql+pymysql',
+    'oracle': 'oracle+cx_oracle',
+    'mssql': 'mssql+pymssql',
+    'firebird': 'firebird+fdb'
+}
+
 
 class BaseAction(Action):
     def __init__(self, config):
@@ -106,6 +114,11 @@ def db_connection(self, kwargs_dict):
         # Get the connection details from either config or from action params
         connection = self.resolve_connection(kwargs_dict)
 
+        # Update Driver with a connector
+        default_driver = DEFAULT_KNOWN_DRIVER_CONNECTORS.get(connection['drivername'], None)
+        if default_driver:
+            connection['drivername'] = default_driver
+
         # Format the connection string
         database_connection_string = URL(**connection)
 

actions/query.yaml

@@ -34,14 +34,6 @@
     drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: false
     query:
       type: string

actions/update.yaml

@@ -34,14 +34,6 @@
     drivername:
       type: string
       description: "Optional override of the database_type in <connection> (required if <connection> is not specified). The type of database that is being connected to."
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: false
     table:
       type: string

config.schema.yaml

@@ -35,13 +35,5 @@ connection:
     drivername:
       description: "The type of database that is being connected to."
       type: string
-      enum:
-        - postgresql
-        - sqlite
-        - mssql
-        - mysql
-        - oracle
-        - firebird
-        - sybase
       required: true
   additionalProperties: false

requirements.txt

@@ -1,6 +1,6 @@
 sqlalchemy
-psycopg2
-mysqlclient
-pyodbc
+psycopg2 <=2.7.5
+pymysql
+pymssql
 cx_Oracle
 fdb