Sync from MySQL to MS SQL normalizations fails #30426
Unanswered
Marcel0643
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
i have some problems setting up a sync from MySQL to MS SQL.
I left everything in the Sources and Destinations Connect at default.
At the Replication i choose Incremental Append
In the Follow is the error log
`
2023-09-14 10:41:45 destination > Destination process done (exit code 0)
2023-09-14 10:41:45 destination > Skipping in-connector normalization
2023-09-14 10:41:45 INFO i.a.w.g.BufferedReplicationWorker(readFromDestination):464 - readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
2023-09-14 10:41:45 INFO i.a.w.t.TemporalAttemptExecution(get):124 - Docker volume job log path: /tmp/workspace/15/0/logs.log
2023-09-14 10:41:45 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.29
2023-09-14 10:41:45 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START DEFAULT NORMALIZATION -----
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:41:45 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):138 - Running with normalization version: airbyte/normalization-mssql:0.4.3
2023-09-14 10:41:45 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/normalization-mssql:0.4.3 exists...
2023-09-14 10:41:46 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/normalization-mssql:0.4.3 was found locally.
2023-09-14 10:41:46 INFO i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = normalization-mssql-normalize-15-0-carfh with resources io.airbyte.config.ResourceRequirements@607110f6[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-09-14 10:41:46 INFO i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/15/0/normalize --log-driver none --name normalization-mssql-normalize-15-0-carfh --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.50.29 airbyte/normalization-mssql:0.4.3 run --integration-type mssql --config destination_config.json --catalog destination_catalog.json
2023-09-14 10:41:47 normalization > Running: transform-config --config destination_config.json --integration-type mssql --out /data/15/0/normalize
2023-09-14 10:41:51 normalization > Namespace(config='destination_config.json', integration_type=<DestinationType.MSSQL: 'mssql'>, out='/data/15/0/normalize')
2023-09-14 10:41:51 normalization > transform_mssql
2023-09-14 10:41:51 normalization > Running: transform-catalog --integration-type mssql --profile-config-dir /data/15/0/normalize --catalog destination_catalog.json --out /data/15/0/normalize/models/generated/ --json-column _airbyte_data
2023-09-14 10:41:51 normalization > Processing destination_catalog.json...
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab1.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab2.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_ctes/dbo/_airbyte_testtabelle_01_ab3.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > Generating airbyte_incremental/dbo/_airbyte_testtabelle_01.sql from _airbyte_testtabelle_01
2023-09-14 10:41:51 normalization > detected no config file for ssh, assuming ssh is off.
2023-09-14 10:41:57 normalization > [--event-buffer-size EVENT_BUFFER_SIZE]
2023-09-14 10:41:57 normalization > --event-buffer-size EVENT_BUFFER_SIZE
2023-09-14 10:41:57 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:41:57 normalization > DBT >=1.0.0 detected; using 10K event buffer size
2023-09-14 10:41:57 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:00 normalization > Running with dbt=1.0.0
2023-09-14 10:42:00 normalization > Partial parse save file not found. Starting full parse.
2023-09-14 10:42:02 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
2023-09-14 10:42:02 normalization > Found 4 models, 0 tests, 0 snapshots, 0 analyses, 621 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2023-09-14 10:42:02 normalization > Concurrency: 8 threads (target='prod')
2023-09-14 10:42:02 normalization > 1 of 1 START incremental model dbo._airbyte_testtabelle_01.............................................................. [RUN]
2023-09-14 10:42:03 normalization > 10:42:03 + "*****".dbo."_airbyte_testtabelle_01"._airbyte_ab_id does not exist yet. The table will be created or rebuilt with dbt.full_refresh
2023-09-14 10:42:03 normalization > Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:03 normalization > 1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
2023-09-14 10:42:03 normalization > Finished running 1 incremental model in 0.98s.
2023-09-14 10:42:03 normalization > Completed with 1 error and 0 warnings:
2023-09-14 10:42:03 normalization > ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:03 normalization > Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
2023-09-14 10:42:03 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:03 normalization > Diagnosing dbt debug to check if destination is available for dbt and well configured (1):
2023-09-14 10:42:03 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:05 normalization > 10:42:05 Running with dbt=1.0.0
2023-09-14 10:42:05 normalization > dbt version: 1.0.0
2023-09-14 10:42:05 normalization > python version: 3.9.9
2023-09-14 10:42:05 normalization > python path: /usr/local/bin/python
2023-09-14 10:42:05 normalization > os info: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
2023-09-14 10:42:05 normalization > Using profiles.yml file at /data/15/0/normalize/profiles.yml
2023-09-14 10:42:05 normalization > Using dbt_project.yml file at /data/15/0/normalize/dbt_project.yml
2023-09-14 10:42:05 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Configuration:
2023-09-14 10:42:06 normalization > profiles.yml file [OK found and valid]
2023-09-14 10:42:06 normalization > dbt_project.yml file [OK found and valid]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Required dependencies:
2023-09-14 10:42:06 normalization > - git [OK found]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Connection:
2023-09-14 10:42:06 normalization > server: *****
2023-09-14 10:42:06 normalization > database: *****
2023-09-14 10:42:06 normalization > schema: dbo
2023-09-14 10:42:06 normalization > port: 1433
2023-09-14 10:42:06 normalization > UID: *****
2023-09-14 10:42:06 normalization > client_id: None
2023-09-14 10:42:06 normalization > authentication: sql
2023-09-14 10:42:06 normalization > encrypt: False
2023-09-14 10:42:06 normalization > trust_cert: False
2023-09-14 10:42:06 normalization > Connection test: [OK connection ok]
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > All checks passed!
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Forward dbt output logs to diagnose/debug errors (0):
2023-09-14 10:42:06 INFO i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 -
2023-09-14 10:42:06 normalization > Running with dbt=1.0.0
2023-09-14 10:42:06 normalization > Partial parse save file not found. Starting full parse.
2023-09-14 10:42:06 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
2023-09-14 10:42:06 normalization > Found 4 models, 0 tests, 0 snapshots, 0 analyses, 621 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2023-09-14 10:42:06 normalization > Concurrency: 8 threads (target='prod')
2023-09-14 10:42:06 normalization > 1 of 1 START incremental model dbo._airbyte_testtabelle_01.............................................................. [RUN]
2023-09-14 10:42:06 normalization > 10:42:03 + "*****".dbo."_airbyte_testtabelle_01"._airbyte_ab_id does not exist yet. The table will be created or rebuilt with dbt.full_refresh
2023-09-14 10:42:06 normalization > Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:06 normalization > 1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
2023-09-14 10:42:06 normalization > Finished running 1 incremental model in 0.98s.
2023-09-14 10:42:06 normalization > Completed with 1 error and 0 warnings:
2023-09-14 10:42:06 normalization > ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
2023-09-14 10:42:06 normalization > Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
2023-09-14 10:42:06 INFO i.a.w.n.DefaultNormalizationRunner(close):207 - Terminating normalization process...
2023-09-14 10:42:06 ERROR i.a.w.g.DefaultNormalizationWorker(run):91 - Normalization failed for job 15.
io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 1)
at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:218) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:89) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-commons-worker-0.50.29.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$2(NormalizationActivityImpl.java:175) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.50.29.jar:?]
at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:124) ~[io.airbyte-airbyte-workers-0.50.29.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-09-14 10:42:06 INFO i.a.w.g.DefaultNormalizationWorker(run):102 - Normalization executed in 20 seconds for job 15.
2023-09-14 10:42:06 INFO i.a.w.g.DefaultNormalizationWorker(run):114 - Normalization summary: io.airbyte.config.NormalizationSummary@7e70e03e[startTime=1694688105595,endTime=1694688126533,failures=[io.airbyte.config.FailureReason@5cd80151[failureOrigin=normalization,failureType=system_error,internalMessage=('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@52835da9[additionalProperties={attemptNumber=0, jobId=15, from_trace_message=true}],stacktrace=AirbyteDbtError:
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),retryable=,timestamp=1694688126531,additionalProperties={}], io.airbyte.config.FailureReason@31ab1050[failureOrigin=normalization,failureType=system_error,internalMessage=('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@4823674b[additionalProperties={attemptNumber=0, jobId=15, from_trace_message=true}],stacktrace=AirbyteDbtError:
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
Unhandled error while executing model.airbyte_utils._airbyte_testtabelle_01
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)")
1 of 1 ERROR creating incremental model dbo._airbyte_testtabelle_01..................................................... [ERROR in 0.25s]
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]'json_value' wird nicht als Name einer integrierten Funktion erkannt. (195) (SQLMoreResults)"),retryable=,timestamp=1694688126531,additionalProperties={}]],additionalProperties={}]
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END DEFAULT NORMALIZATION -----
2023-09-14 10:42:06 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-09-14 10:42:06 INFO i.a.w.t.s.a.AppendToAttemptLogActivityImpl(log):56 - Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=0, totalCompleteFailures=0, successivePartialFailures=1, totalPartialFailures=1)
Backoff before next attempt: 0 seconds
`
Beta Was this translation helpful? Give feedback.
All reactions