IBM watsonx.data destination connector: add max connection retries, increase retry defaults and limits (#600)

Author: Paul-Cornell

snippets/destination_connectors/ibm_watsonxdata.sh.mdx

@@ -23,6 +23,7 @@ unstructured-ingest \
     --catalog $IBM_ICEBERG_CATALOG \
     --namespace $IBM_ICEBERG_SCHEMA \
     --table $IBM_ICEBERG_TABLE \
-    --max-retries 5 \
+    --max-retries 50 \
+    --max-retries-connection 10 \
     --record-id-key $IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN
 ```

snippets/destination_connectors/ibm_watsonxdata.v2.py.mdx

@@ -56,7 +56,8 @@ if __name__ == "__main__":
         uploader_config=IbmWatsonxUploaderConfig(
             namespace=os.getenv("IBM_ICEBERG_SCHEMA"),
             table=os.getenv("IBM_ICEBERG_TABLE"),
-            max_retries=5,
+            max_retries=50,
+            max_retries_connection=10,
             record_id_key=os.getenv("IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN")
         )
     ).run()

snippets/destination_connectors/ibm_watsonxdata_rest_create.mdx

@@ -19,6 +19,7 @@ curl --request 'POST' --location \
         "namespace": "<namespace>",
         "table": "<table>",
         "max_retries": <max-retries>,
+        "max_retries_connection": <max-retries-connection>,
         "record_id_key": "<record-id-key>"
     }
 }'

snippets/destination_connectors/ibm_watsonxdata_sdk.mdx

@@ -26,6 +26,7 @@ with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as clien
                     namespace="<namespace>",
                     table="<table>",
                     max_retries=<max-retries>,
+                    max_retries_connection=<max-retries-connection>,
                     record_id_key="<record-id-key>"
                 )
             )

snippets/general-shared-text/ibm-watsonxdata-api-placeholders.mdx

@@ -8,5 +8,6 @@
 - `<catalog>` (_required_): The name of the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance.
 - `<namespace>` (_required_): The name of the target namespace (also known as a schema) within the catalog.
 - `<table>` (_required_): The name of the target table within the namespace (schema).
-- `<max-retries>`: The maximum number of retries for the upload process. The default is `5`.
+- `<max-retries>`: The maximum number of retries for the upload process. The default is `50`. If specified, it must be a number between `2` and `500`, inclusive.
+- `<max-retries-connection>`: The maximum number of retries when connecting to the catalog. The default is `10`. If specified, it must be a number between `2` and `100`, inclusive.
 - `<record-id-key>`: The name of the column that uniquely identifies each record in the target table. The default is `record_id`.

snippets/general-shared-text/ibm-watsonxdata-cli-api.mdx

@@ -21,4 +21,7 @@ The following environment variables:
 - `IBM_ICEBERG_TABLE` - The name of the target table in the target schema, represented by `--table` (CLI) or `table` (Python).
 - `IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN` - The name of the column that uniquely identifies each record in the target table, represented by `--record-id-key` (CLI) or `record_id_key` (Python). The default is `record_id`.
 
-Additionally, `--max-retries` (CLI) or `max_retries` (Python) is an optional parameter that specifies the number of times to retry uploading data. The default is `5`. If specified, it must be a number between `2` and `10`, inclusive.
+Additionally:
+
+- `--max-retries-connection` (CLI) or `max_retries_connection` (Python) is an optional parameter that specifies the maximum number of retries when connecting to the catalog. The default is `10`. If specified, it must be a number between `2` and `100`, inclusive.
+- `--max-retries` (CLI) or `max_retries` (Python) is an optional parameter that specifies the number of times to retry uploading data. The default is `50`. If specified, it must be a number between `2` and `500`, inclusive.

snippets/general-shared-text/ibm-watsonxdata-platform.mdx

@@ -10,5 +10,6 @@ Fill in the following fields:
 - **Catalog** (_required_): The name of the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance.
 - **Namespace** (_required_): The name of the target namespace (also known as a schema) within the catalog.
 - **Table** (_required_): The name of the target table within the namespace (schema).
-- **Max Retries**: The maximum number of retries for the upload process. The default is `5`.
+- **Max Connection Retries**: The maximum number of retries when connecting to the catalog. The default is `10`. If specified, it must be a number between `2` and `100`, inclusive.
+- **Max Retries**: The maximum number of retries when uploading data. The default is `50`. If specified, it must be a number between `2` and `500`, inclusive.
 - **Record ID Key**: The name of the column that uniquely identifies each record in the target table. The default is `record_id`.