test: CI/CD

Initial version with schemaless

Initial version with schemaless

add partition when schemaless

black

initial commit with infer schema

initial commit with infer schema : handle json msg not dict

initial commit with infer schema : handle json msg not dict

initial commit with infer schema : one generator by row

initial commit with infer schema : one generator by row

rollback infer schema and clean integration of schemaless process

remove unused var

initial commit with infer_schema

limit errors count

add schemaless and infer_schema doc

fix index out of range

reduce info messages

fix python format

Author: lideke

README.md

@@ -154,8 +154,11 @@ sample [target-config.json](/sample_config/target-config-exchange-rates-api.json
     * `truncate`: Deleting all previous rows and uploading the new ones to the table
     * `incremental`: **Upserting** new rows into the table, using the **primary key** given by the tap connector
       (if it finds an old row with same key, updates it. Otherwise it inserts the new row)
+    * `schemaless` create table with unique column `record` this parameter is usful in case data source is schemaless like `mongodb`
+    * `infer_schema` this parameter create new tables `_sdc_schema` with infered schema from input records : this is a better alternative to schema received from tap
  - WARNING: We do not recommend using `incremental` option (which uses `MERGE` SQL statement). It might result in loss of production data, because historical records get updated. Instead, we recommend using the `append` replication method, which will preserve historical data. 
 
+
 Sample **target-config.json** file:
 
 ```
@@ -166,7 +169,9 @@ Sample **target-config.json** file:
     "validate_records": true,
     "add_metadata_columns": true,
     "location": "EU",
-    "table_config": "target-tables-config.json"
+    "table_config": "target-tables-config.json",
+    "schemaless": true,
+    "infer_schema": true
 }
 ```
 

requirements.txt

@@ -1,4 +1,5 @@
 singer-python~=5.12.2
 google-cloud-bigquery~=3.3.5
 jsonschema~=2.6.0
-setuptools~=65.5.0
+setuptools~=65.5.0
+bigquery_schema_generator==1.4.1

target_bigquery/__init__.py

@@ -74,6 +74,8 @@ def main():
     location = config.get("location", "US")
     validate_records = config.get("validate_records", True)
     add_metadata_columns = config.get("add_metadata_columns", True)
+    schemaless = config.get("schemaless", False)
+    infer_schema = config.get("infer_schema", False)
 
     # we can pass merge state option via CLI param
     merge_state_messages_cli = flags.merge_state_messages
@@ -125,6 +127,8 @@ def main():
             table_prefix=table_prefix,
             table_suffix=table_suffix,
             add_metadata_columns=add_metadata_columns,
+            schemaless=schemaless,
+            infer_schema=infer_schema,
             table_configs=table_configs,
             max_cache=max_cache
         )

target_bigquery/process.py

@@ -56,7 +56,7 @@ def process(
                 yield s
 
         elif isinstance(msg, singer.StateMessage):
-            logger.info("Updating state with {}".format(msg.value))
+            logger.debug("Updating state with {}".format(msg.value))
             for s in handler.handle_state_message(msg):
                 logger.info(f"Pushing state: {s}")
                 yield s