Getting error on schema discovery.

[tolemac]

Hi all!

Airbyte version: 0.61.0
OS Version / Instance: Ubuntu 22.04.4 LTS
Deployment: Docker
Source Connector and version: SQL Server 4.0.27
Destination Connector and version: Postgress 2.0.10
Step where error happened: Setup new connection

I'm begining to test airbyte, I deployed it using docker compose following this instructions: https://docs.airbyte.com/deploying-airbyte/docker-compose

I could transfer data from source database to dest database but when I try to move a big set of data I get memory and cpu issues. The machine where I were testing it was insuficient.

Ok, I create a new machine with more resources (CPU, RAM and HDD), install ubuntu 22.04 and docker.
However now when I try to move data between 2 databases I get this error:

And there is the log:

2024-06-05 10:20:23 �[46mplatform�[0m > Docker volume job log path: /tmp/workspace/84858c58-250b-4e21-a53f-2d01b8de0618/0/logs.log
2024-06-05 10:20:23 �[46mplatform�[0m > Executing worker wrapper. Airbyte version: 0.61.0
2024-06-05 10:20:23 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-06-05 10:20:23 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-06-05 10:20:23 �[46mplatform�[0m > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-06-05 10:20:23 �[46mplatform�[0m > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-06-05 10:20:23 �[46mplatform�[0m > Checking if airbyte/source-mssql:4.0.27 exists...
2024-06-05 10:20:23 �[46mplatform�[0m > airbyte/source-mssql:4.0.27 was found locally.
2024-06-05 10:20:23 �[46mplatform�[0m > Creating docker container = source-mssql-discover-84858c58-250b-4e21-a53f-2d01b8de0618-0-xoizb with resources io.airbyte.config.ResourceRequirements@380b86d2[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-06-05 10:20:23 �[46mplatform�[0m > Preparing command: docker run --rm --init -i -w /data/84858c58-250b-4e21-a53f-2d01b8de0618/0 --log-driver none --name source-mssql-discover-84858c58-250b-4e21-a53f-2d01b8de0618-0-xoizb --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-mssql:4.0.27 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.61.0 -e WORKER_JOB_ID=84858c58-250b-4e21-a53f-2d01b8de0618 airbyte/source-mssql:4.0.27 discover --config source_config.json
2024-06-05 10:20:23 �[46mplatform�[0m > Reading messages from protocol version 0.2.0
2024-06-05 10:20:24 �[46mplatform�[0m > INFO main i.a.i.s.m.MssqlSource(main):619 starting source: class io.airbyte.integrations.source.mssql.MssqlSource
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main i.a.c.i.b.IntegrationCliParser$Companion(parseOptions):144 integration args: {discover=null, config=source_config.json}
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main i.a.c.i.b.IntegrationRunner(runInternal):124 Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main i.a.c.i.b.IntegrationRunner(runInternal):125 Command: DISCOVER
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main i.a.c.i.b.IntegrationRunner(runInternal):126 Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-06-05 10:20:25 �[46mplatform�[0m > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-06-05 10:20:25 �[46mplatform�[0m > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-06-05 10:20:25 �[46mplatform�[0m > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):423 Starting connection with method: NO_TUNNEL
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-1 - Starting...
2024-06-05 10:20:25 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-1 - Start completed.
2024-06-05 10:20:26 �[46mplatform�[0m > INFO main i.a.c.i.s.j.AbstractJdbcSource(discoverInternal):291 Internal schemas to exclude: [spt_fallback_db, spt_monitor, cdc, spt_values, INFORMATION_SCHEMA, spt_fallback_usg, MSreplication_options, sys, spt_fallback_dev]
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main i.a.c.i.s.j.AbstractJdbcSource(discoverPrimaryKeys):428 Discover primary keys for tables: [MUNICIPIO, PROVINCIA, USUARIO]
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(close):349 HikariPool-1 - Shutdown initiated...
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(close):351 HikariPool-1 - Shutdown completed.
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-2 - Starting...
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-2 - Start completed.
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main i.a.c.i.b.IntegrationRunner(runInternal):268 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-06-05 10:20:27 �[46mplatform�[0m > INFO main i.a.i.s.m.MssqlSource(main):621 completed source: class io.airbyte.integrations.source.mssql.MssqlSource

I don't know what can be the problem ...
I don't know how to see more logs or view the error description, I don't know what happen ...

Can some one help me?

[ledinhthanh]

same issue

[tolemac]

Comparing the logs, between first that work fine and second that fails, I can see that the second starting a second HikariPool, called HikariPool-2, the first one end when HikariPool-1 shutdowns.

In addition I can see, in the first, a message when start the log that says Attempt 0 to save workflow id for cancellation and at the end Attempt 0 to call to write discover schema result. These messages not are present in the second.

[tolemac]

That is independent of source type, I have tried MSSQL Server as source and Postgres too, the same result.

I think is not source connector problem, I think is a main machine communication problem with source machine, however the creation of source success, the source connection testing is successful ...

I dont understand ...

[tolemac]

I couldn't solve it.

I have created a new virtual machine, I have make a fresh Ubuntu 24.04 and Docker installation and I have test it again and voila, it works.

But I don't know what is the reason.