Merge branch 'develop'

Author: dolfinus

.github/workflows/data/db/tracked.txt

@@ -1,5 +1,6 @@
 .github/workflows/data/db/**
-onetl/db_connection/db_connection.py
-onetl/db_connection/jdbc*.py
+onetl/db_connection/db_connection/*
 onetl/db_connection/dialect_mixins/*
+onetl/db_connection/jdbc_connection/*
+onetl/db_connection/jdbc_mixin/*
 onetl/db/**

.pre-commit-config.yaml

@@ -52,7 +52,7 @@ repos:
           - --no-extra-eol
 
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.6
+    rev: v2.3.0
     hooks:
       - id: codespell
         args: [-w]

README.rst

@@ -341,7 +341,7 @@ Read data from MSSQL, transform & write to Hive.
         database="Telecom",
         spark=spark,
         # These options are passed to MSSQL JDBC Driver:
-        extra={"ApplicationIntent": "ReadOnly"},
+        extra={"applicationIntent": "ReadOnly"},
     ).check()
 
     # >>> INFO:|MSSQL| Connection is available

docs/changelog/0.11.0.rst

@@ -28,41 +28,6 @@ most of users will not see any differences.
 
   This brings few bugfixes with datetime format handling.
 
-- Serialize ``ColumnDatetimeHWM`` to Clickhouse's ``DateTime64(6)`` (precision up to microseconds) instead of ``DateTime`` (precision up to seconds)  (:github:pull:`267`).
-
-  In previous onETL versions, ``ColumnDatetimeHWM`` value was rounded to the second, and thus reading some rows that were read in previous runs,
-  producing duplicates.
-
-  For Clickhouse versions below 21.1 comparing column of type ``DateTime`` with a value of type ``DateTime64`` is not supported, returning an empty dataframe.
-  To avoid this, replace:
-
-  .. code:: python
-
-    DBReader(
-        ...,
-        hwm=DBReader.AutoDetectHWM(
-            name="my_hwm",
-            expression="hwm_column",  # <--
-        ),
-    )
-
-  with:
-
-  .. code:: python
-
-    DBReader(
-        ...,
-        hwm=DBReader.AutoDetectHWM(
-            name="my_hwm",
-            expression="CAST(hwm_column AS DateTime64)",  # <-- add explicit CAST
-        ),
-    )
-
-- Pass JDBC connection extra params as ``properties`` dict instead of URL with query part (:github:pull:`268`).
-
-  This allows passing custom connection parameters like ``Clickhouse(extra={"custom_http_options": "option1=value1,option2=value2"})``
-  without need to apply urlencode to parameter value, like ``option1%3Dvalue1%2Coption2%3Dvalue2``.
-
 - For JDBC connections add new ``SQLOptions`` class for ``DB.sql(query, options=...)`` method (:github:pull:`272`).
 
   Firsly, to keep naming more consistent.
@@ -166,6 +131,41 @@ most of users will not see any differences.
   For now, ``DB.fetch(query, options=...)`` and ``DB.execute(query, options=...)`` can accept ``JDBCOptions``,  to keep backward compatibility,
   but emit a deprecation warning. The old class will be removed in ``v1.0.0``.
 
+- Serialize ``ColumnDatetimeHWM`` to Clickhouse's ``DateTime64(6)`` (precision up to microseconds) instead of ``DateTime`` (precision up to seconds)  (:github:pull:`267`).
+
+  In previous onETL versions, ``ColumnDatetimeHWM`` value was rounded to the second, and thus reading some rows that were read in previous runs,
+  producing duplicates.
+
+  For Clickhouse versions below 21.1 comparing column of type ``DateTime`` with a value of type ``DateTime64`` is not supported, returning an empty dataframe.
+  To avoid this, replace:
+
+  .. code:: python
+
+    DBReader(
+        ...,
+        hwm=DBReader.AutoDetectHWM(
+            name="my_hwm",
+            expression="hwm_column",  # <--
+        ),
+    )
+
+  with:
+
+  .. code:: python
+
+    DBReader(
+        ...,
+        hwm=DBReader.AutoDetectHWM(
+            name="my_hwm",
+            expression="CAST(hwm_column AS DateTime64)",  # <-- add explicit CAST
+        ),
+    )
+
+- Pass JDBC connection extra params as ``properties`` dict instead of URL with query part (:github:pull:`268`).
+
+  This allows passing custom connection parameters like ``Clickhouse(extra={"custom_http_options": "option1=value1,option2=value2"})``
+  without need to apply urlencode to parameter value, like ``option1%3Dvalue1%2Coption2%3Dvalue2``.
+
 Features
 --------
 
@@ -188,14 +188,14 @@ Improve user experience with Kafka messages and Database tables with serialized
     * ``CSV.parse_column(col, schema=...)`` (:github:pull:`258`).
     * ``XML.parse_column(col, schema=...)`` (:github:pull:`269`).
 
-   This allows parsing data in ``value`` field of Kafka message or string/binary column of some table as a nested Spark structure.
+  This allows parsing data in ``value`` field of Kafka message or string/binary column of some table as a nested Spark structure.
 
 - Add ``FileFormat.serialize_column(...)`` method to several classes:
-    * ``Avro.serialize_column(col)`` (:github:pull:`265`).
-    * ``JSON.serialize_column(col)`` (:github:pull:`257`).
-    * ``CSV.serialize_column(col)`` (:github:pull:`258`).
+   * ``Avro.serialize_column(col)`` (:github:pull:`265`).
+   * ``JSON.serialize_column(col)`` (:github:pull:`257`).
+   * ``CSV.serialize_column(col)`` (:github:pull:`258`).
 
-   This allows saving Spark nested structures or arrays to ``value`` field of Kafka message or string/binary column of some table.
+  This allows saving Spark nested structures or arrays to ``value`` field of Kafka message or string/binary column of some table.
 
 Improvements
 ------------
@@ -220,7 +220,7 @@ Few documentation improvements.
 
 - Add note about connecting to Clickhouse cluster. (:github:pull:`280`).
 
-- Add notes about versions when specific class/method/attribute/argument was added, renamed or changed behavior (:github:`282`).
+- Add notes about versions when specific class/method/attribute/argument was added, renamed or changed behavior (:github:pull:`282`).
 
 
 Bug Fixes

docs/changelog/0.11.1.rst

@@ -0,0 +1,13 @@
+0.11.1 (2024-05-29)
+===================
+
+Features
+--------
+
+- Change ``MSSQL.port`` default from ``1433`` to ``None``, allowing use of ``instanceName`` to detect port number. (:github:pull:`287`)
+
+
+Bug Fixes
+---------
+
+- Remove ``fetchsize`` from ``JDBC.WriteOptions``. (:github:pull:`288`)

docs/changelog/index.rst

@@ -3,6 +3,7 @@
     :caption: Changelog
 
     DRAFT
+    0.11.1
     0.11.0
     0.10.2
     0.10.1

docs/connection/db_connection/hive/write.rst

@@ -21,8 +21,15 @@ Examples
     # Use the Hive partitioning columns to group the data. Create only 20 files per Hive partition.
     # Also sort the data by column which most data is correlated with, reducing files size.
     write_df = df.repartition(
-        20, "country", "business_date", "user_id"
-    ).sortWithinPartitions("country", "business_date", "user_id")
+        20,
+        "country",
+        "business_date",
+        "user_id",
+    ).sortWithinPartitions(
+        "country",
+        "business_date",
+        "user_id",
+    )
 
     writer = DBWriter(
         connection=hive,

docs/connection/db_connection/kafka/prerequisites.rst

@@ -46,6 +46,7 @@ Authentication mechanism
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 Kafka can support different authentication mechanism (also known as `SASL <https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer>`_).
+
 List of currently supported mechanisms:
   * :obj:`PLAIN <onetl.connection.db_connection.kafka.kafka_basic_auth.KafkaBasicAuth>`. To no confuse this with ``PLAINTEXT`` connection protocol, onETL uses name ``BasicAuth``.
   * :obj:`GSSAPI <onetl.connection.db_connection.kafka.kafka_kerberos_auth.KafkaKerberosAuth>`. To simplify naming, onETL uses name ``KerberosAuth``.

docs/connection/db_connection/mssql/prerequisites.rst

@@ -27,9 +27,11 @@ Connecting to MSSQL
 Connection port
 ~~~~~~~~~~~~~~~
 
-Connection is usually performed to port 1443. Port may differ for different MSSQL instances.
+Connection is usually performed to port 1433. Port may differ for different MSSQL instances.
 Please ask your MSSQL administrator to provide required information.
 
+For named MSSQL instances (``instanceName`` option), `port number is optional <https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-ver16#named-and-multiple-sql-server-instances>`_, and could be omitted.
+
 Connection host
 ~~~~~~~~~~~~~~~
 

docs/connection/db_connection/mssql/types.rst

@@ -35,7 +35,7 @@ This is how MSSQL connector performs this:
     it will be populated by MSSQL.
 
 .. [2]
-    This is true only if DataFrame column is a ``StringType()``, because text value is parsed automatically to tagret column type.
+    This is true only if DataFrame column is a ``StringType()``, because text value is parsed automatically to target column type.
 
     But other types cannot be silently converted, like ``int -> text``. This requires explicit casting, see `DBWriter`_.