Merge pull request #2625 from creideiki/mysql-mixin

Add MySQL/MariaDB support to the SQL mixin

Author: sebix

CHANGELOG.md

@@ -18,6 +18,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 ### Core
 - Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
 - `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
+- `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
 
 ### Development
 
@@ -45,6 +46,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
   - Add new parameter `templating` for additional template variables.
   - Add new parameter `allowed_fieldnames` for csv field specification.
   - Add new parameter `fieldnames_translation` for naming csv headers (PR#2610 by Lukas Heindl, fixes #2586).
+- `intelmq.bots.outputs.sql.output`: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
 
 ### Documentation
 - Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)

docs/user/bots.md

@@ -2905,8 +2905,7 @@ Order of operation: `strip -> replace -> split`. These three methods can be comb
 
 ### Generic DB Lookup <div id="intelmq.bots.experts.generic_db_lookup.expert" />
 
-This bot is capable for enriching intelmq events by lookups to a database. Currently only PostgreSQL and SQLite are
-supported.
+This bot is capable for enriching intelmq events by lookups to a database. Currently PostgreSQL, SQLite, MSSQL, and MySQL/MariaDB are supported.
 
 If more than one result is returned, a ValueError is raised.
 
@@ -2918,7 +2917,7 @@ If more than one result is returned, a ValueError is raised.
 
 **`engine`**
 
-(required, string) Allowed values: `postgresql` or `sqlite`.
+(required, string) Allowed values: `postgresql`, `sqlite`, `mssql`, or `mysql`.
 
 **`database`**
 
@@ -2928,23 +2927,25 @@ If more than one result is returned, a ValueError is raised.
 
 (optional, string) Name of the table. Defaults to `contacts`.
 
-*PostgreSQL specific parameters*
+*Database server (i.e. not SQLite) specific parameters*
 
 **`host`**
 
-(optional, string) Hostname of the PostgreSQL server. Defaults to `localhost`.
+(optional, string) Hostname of the database server. Defaults to `localhost`.
 
 **`port`**
 
-(optional, integer) Port of the PostgreSQL server. Defaults to 5432.
+(optional, integer) Port of the database server. Defaults to 5432 (which is the default for PostgreSQL).
 
 **`user`**
 
-(optional, string) Username for accessing PostgreSQL. Defaults to `intelmq`.
+(optional, string) Username for accessing the database server. Defaults to `intelmq`.
 
 **`password`**
 
-(optional, string) Password for accessing PostgreSQL. Defaults to ?.
+(optional, string) Password for accessing the database server. Defaults to ?.
+
+*PostgreSQL specific parameters*
 
 **`sslmode`**
 
@@ -5294,7 +5295,7 @@ Client certificates are not supported. If `http_verify_cert` is true, TLS certif
 
 ### SQL <div id="intelmq.bots.outputs.sql.output" />
 
-SQL is the bot responsible to send events to a PostgreSQL, SQLite, or MSSQL Database.
+SQL is the bot responsible to send events to a PostgreSQL, SQLite, MSSQL, or MySQL/MariaDB database.
 
 !!! note
     When activating autocommit, transactions are not used. See: <http://initd.org/psycopg/docs/connection.html#connection.autocommit>
@@ -5311,7 +5312,7 @@ The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Chec
 
 **`engine`**
 
-(required, string) Allowed values are `postgresql`, `sqlite`, or `mssql`.
+(required, string) Allowed values are `postgresql`, `sqlite`, `mssql`, or `mysql`.
 
 **`database`**
 
@@ -5343,7 +5344,7 @@ The parameters marked with 'PostgreSQL' will be sent to libpq via psycopg2. Chec
 
 **`sslmode`**
 
-(optional, string) Database sslmode, Allowed values: `disable`, `allow`, `prefer`, `require`, `verify-ca` or `verify-full`. See: <https://www.postgresql.org/docs/current/static/images/libpq-connect.html#libpq-connect-sslmode>. Defaults to `require`.
+(optional, string, PostgreSQL only) Database sslmode, Allowed values: `disable`, `allow`, `prefer`, `require`, `verify-ca` or `verify-full`. See: <https://www.postgresql.org/docs/current/static/images/libpq-connect.html#libpq-connect-sslmode>. Defaults to `require`.
 
 **`table`**
 

intelmq/bots/outputs/sql/REQUIREMENTS.txt

@@ -3,3 +3,4 @@
 
 psycopg2-binary>=2.5.5
 pymssql>=2.2
+PyMySQL>=0.10

intelmq/bots/outputs/sql/output.py

@@ -24,7 +24,7 @@ def g(obj):
 
 
 class SQLOutputBot(OutputBot, SQLMixin):
-    """Send events to a PostgreSQL or SQLite database"""
+    """Send events to an SQL database"""
     autocommit = True
     database = "intelmq-events"
     engine = None

intelmq/lib/mixins/sql.py

@@ -16,12 +16,13 @@ class SQLMixin:
     You do not have to bother:
     * connecting database in the self.init() method, just call super().init(), self.cur will be set
     * catching exceptions, just call self.execute() instead of self.cur.execute()
-    * self.format_char will be set to '%s' in PostgreSQL and to '?' in SQLite
+    * self.format_char will be set to '?' in SQLite and '%s' otherwise
     """
 
     POSTGRESQL = "postgresql"
     SQLITE = "sqlite"
     MSSQL = "mssql"
+    MYSQL = "mysql"
     _default_engine = "postgresql"
     engine = None
     # overwrite the default value from the OutputBot
@@ -40,7 +41,8 @@ def _init_sql(self):
         self._engine_name = getattr(self, 'engine', self._default_engine).lower()
         engines = {SQLMixin.POSTGRESQL: (self._init_postgresql, "%s"),
                    SQLMixin.SQLITE: (self._init_sqlite, "?"),
-                   SQLMixin.MSSQL: (self._init_mssql, "%s")}
+                   SQLMixin.MSSQL: (self._init_mssql, "%s"),
+                   SQLMixin.MYSQL: (self._init_mysql, "%s")}
         for key, val in engines.items():
             if self._engine_name == key:
                 val[0]()
@@ -110,6 +112,23 @@ def _init_mssql(self):
                        },
                       autocommitable=True)
 
+    def _init_mysql(self):
+        try:
+            import pymysql
+        except ImportError:
+            raise exceptions.MissingDependencyError("pymysql")
+
+        self._connect(pymysql,
+                      {"database": self.database,
+                       "user": self.user,
+                       "password": self.password,
+                       "host": self.host,
+                       "port": self.port,
+                       "connect_timeout": getattr(self, 'connect_timeout', 5)
+                       },
+                      autocommitable=True)
+        self.cur.execute("SET sql_mode = CONCAT_WS(',', (SELECT @@sql_mode), 'ANSI_QUOTES')")
+
     def execute(self, query: str, values: tuple, rollback=False):
         try:
             self.logger.debug('Executing %r.', (query, values))