Issue with setting up sources in Airbyte running on EC2

[avaradha]

Hi
I have setup an Airbyte instance on EC2 of type m5.large which runs Amazon Linux 2023 linux. The EC2 instance is behind a squid proxy. Have setup the proxy setting in docker config and I was able to bring Airbyte Up. But when I try to setup Google Sheet as a source, it goes through the check but fails with this error: Unknown error (http.504.sGrcbYgB4vyx1R9WiqgGGA). The worker log shows this error message:

java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:172) ~[?:?]
        at java.io.BufferedInputStream.implRead(BufferedInputStream.java:373) ~[?:?]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:367) ~[?:?]
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:333) ~[?:?]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:376) ~[?:?]
        at sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:219) ~[?:?]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:173) ~[?:?]
        at java.io.InputStreamReader.read(InputStreamReader.java:189) ~[?:?]
        at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:?]
        at java.io.BufferedReader.implReadLine(BufferedReader.java:371) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:348) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:437) ~[?:?]
        at io.airbyte.workers.WorkerUtils.getStdErrFromErrorStream(WorkerUtils.java:248) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:265) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.50.11.jar:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]

Not sure if I had missed something. Can anyone help on how to debug this issue?

[avaradha]

This is the complete log message for the check:

2023-08-10 13:49:29 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2023-08-10 13:49:29 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2023-08-10 13:49:29 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-08-10 13:49:29 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK -----
2023-08-10 13:49:29 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-08-10 13:49:29 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2023-08-10 13:49:29 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2023-08-10 13:49:29 INFO i.a.c.EnvConfigs(getEnvOrDefault):1235 - Using default value for environment variable LAUNCHDARKLY_KEY: ''
2023-08-10 13:49:29 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-google-sheets:0.3.1 exists...
2023-08-10 13:49:29 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-google-sheets:0.3.1 was found locally.
2023-08-10 13:49:29 INFO i.a.w.p.DockerProcessFactory(create):142 - Creating docker container = source-google-sheets-check-fbc358e6-ed2a-4ca0-bb0b-beb779ca475c-0-wtmgo with resources io.airbyte.config.ResourceRequirements@4bf71232[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2023-08-10 13:49:29 INFO i.a.w.p.DockerProcessFactory(create):195 - Preparing command: docker run --rm --init -i -w /data/fbc358e6-ed2a-4ca0-bb0b-beb779ca475c/0 --log-driver none --name source-google-sheets-check-fbc358e6-ed2a-4ca0-bb0b-beb779ca475c-0-wtmgo --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.3.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.11 -e WORKER_JOB_ID=fbc358e6-ed2a-4ca0-bb0b-beb779ca475c airbyte/source-google-sheets:0.3.1 check --config source_config.json
2023-08-10 13:49:29 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):177 - Reading messages from protocol version 0.2.0
2023-08-10 13:54:39 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):231 - Running sync worker cancellation...
2023-08-10 13:54:39 WARN i.a.w.g.DefaultCheckConnectionWorker(run):110 - Check connection job subprocess finished with exit code 143
2023-08-10 13:54:39 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):235 - Interrupting worker thread...
2023-08-10 13:54:39 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$6):238 - Cancelling completable future...
2023-08-10 13:54:39 WARN i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):60 - Job either timed out or was cancelled.
2023-08-10 13:54:39 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):128 - Lost connection to the source:
java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:172) ~[?:?]
        at java.io.BufferedInputStream.implRead(BufferedInputStream.java:373) ~[?:?]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:367) ~[?:?]
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:333) ~[?:?]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:376) ~[?:?]
        at sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:219) ~[?:?]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:173) ~[?:?]
        at java.io.InputStreamReader.read(InputStreamReader.java:189) ~[?:?]
        at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:?]
        at java.io.BufferedReader.implReadLine(BufferedReader.java:371) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:348) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:437) ~[?:?]
        at io.airbyte.workers.WorkerUtils.getStdErrFromErrorStream(WorkerUtils.java:248) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:265) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.50.11.jar:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-08-10 13:54:39 INFO i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
2023-08-10 13:54:39 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=472afadf-587b-3942-919b-cb229356852f, activityType=RunWithJobOutput, attempt=1
java.util.concurrent.CancellationException: null
        at java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2510) ~[?:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$6(TemporalAttemptExecution.java:241) ~[io.airbyte-airbyte-workers-0.50.11.jar:?]
        at io.airbyte.commons.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:59) ~[io.airbyte-airbyte-commons-temporal-0.50.11.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$7(TemporalAttemptExecution.java:244) ~[io.airbyte-airbyte-workers-0.50.11.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-08-10 13:54:39 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Lost connection to the source:
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:130) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.50.11.jar:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:172) ~[?:?]
        at java.io.BufferedInputStream.implRead(BufferedInputStream.java:373) ~[?:?]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:367) ~[?:?]
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:333) ~[?:?]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:376) ~[?:?]
        at sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:219) ~[?:?]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:173) ~[?:?]
        at java.io.InputStreamReader.read(InputStreamReader.java:189) ~[?:?]
        at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:?]
        at java.io.BufferedReader.implReadLine(BufferedReader.java:371) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:348) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:437) ~[?:?]
        at io.airbyte.workers.WorkerUtils.getStdErrFromErrorStream(WorkerUtils.java:248) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.WorkerUtils.throwWorkerException(WorkerUtils.java:265) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:120) ~[io.airbyte-airbyte-commons-worker-0.50.11.jar:?]
        ... 3 more
2023-08-10 13:54:39 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-08-10 13:54:39 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK -----