Normalization failing: cannot understand why in the logs. Need your help

[judiennetSC]

Hi everyone,

I'm encountering an issue while trying to load data from a source (Chargebee) to a destination (PostgreSQL) - I don't believe these are important.
The problem appears at the default normalization step (see the logs below). Temporal seems concerned too?

My setup on AWS:
Instance type: EC2, t4g.large
OS: Amazon Linux 2023 AMI (ebs, hvm, kernel-6.1)
Architecture: arm64
Docker version: 20.10
Docker Compose version: 2.21
Airbyte version: 0.50.30

Here are the logs below. Can you help me please?
Thank you!

5ca40492_8291_4d81_83cb_6e375b590f60_job_1_attempt_1_txt.txt

5ca40492_8291_4d81_83cb_6e375b590f60_job_1_attempt_7_txt.txt

[tomt00001]

Did you ever find a solution? I am having the same error after following the directions here

Feels like the container that runs normalization isn't running or getting run.

[tomt00001]

I found this

exec format error is an error you get when you try to run a binary for architecture X on a machine with architecture Y, and the machine is not set up to translate CPU instructions from X to Y.

Maybe something to do with aarch64 architecture?

[judiennetSC]

Maybe! But I ran it well on my Mac Apple M1 (arm64 too).
Did you try basic architecture x86 and it worked @tomt00001 ?

[tomt00001]

Funny you should mention x86 was our next thing to try but I have not done that yet. I also did all the testing on my M1 also.

The container listed in the error message is normalization. According to the docker in the ec2 instance, I do have the latest image

[tomt00001]

@judiennetSC I have an update.

tl;dr for the future, changed from t4g.large to t2.large and followed same installation steps, works!

Summary of my setup
Instance type: EC2, t2.large
OS: Amazon Linux 2023 AMI (al2023-ami-2023.2.20230920.1-kernel-6.1-x86_64)
Architecture: x86_64
Docker version: 24.0.5
Docker compose version: v2.22.0
Airbyte version: 0.50.30

The exec format error that happens on t4g-large (aarm64) feels like it is related to the Graviton platform and airbyte. The actual error seems like it is having trouble spawning the airbyte normalization container when it's on Graviton. This does work locally on my MacBook Pro 16-inch, 2021 with Apple M1 Pro CPU

Log messages generated inside the Graviton EC2 instance (similar to @judiennetSC's error messages)

2023-09-25 20:45:09 INFO i.a.w.t.TemporalAttemptExecution(get):124 - Docker volume job log path: /tmp/workspace/7/4/logs.log 2023-09-25 20:45:09 INFO i.a.w.t.TemporalAttemptExecution(get):129 - Executing worker wrapper. Airbyte version: 0.50.30 2023-09-25 20:45:09 INFO i.a.a.c.AirbyteApiClient(retryWithJitterThrows):290 - Attempt 0 to save workflow id for cancellation 2023-09-25 20:45:09 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):138 - Running with normalization version: airbyte/normalization:0.4.3 2023-09-25 20:45:09 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-09-25 20:45:09 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START DEFAULT NORMALIZATION ----- 2023-09-25 20:45:09 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-09-25 20:45:09 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/normalization:0.4.3 exists... 2023-09-25 20:45:09 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/normalization:0.4.3 was found locally. 2023-09-25 20:45:09 INFO i.a.w.p.DockerProcessFactory(create):143 - Creating docker container = normalization-normalize-7-4-xvmtz with resources io.airbyte.config.ResourceRequirements@66368b0[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null 2023-09-25 20:45:09 INFO i.a.w.p.DockerProcessFactory(create):196 - Preparing command: docker run --rm --init -i -w /data/7/4/normalize --log-driver none --name normalization-normalize-7-4-xvmtz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.50.30 airbyte/normalization:0.4.3 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json 2023-09-25 20:45:09 INFO i.a.w.n.DefaultNormalizationRunner(close):207 - Terminating normalization process... 2023-09-25 20:45:09 ERROR i.a.w.g.DefaultNormalizationWorker(run):91 - Normalization failed for job 7. io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 1) at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:218) ~[io.airbyte-airbyte-commons-worker-0.50.30.jar:?] at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:89) ~[io.airbyte-airbyte-commons-worker-0.50.30.jar:?] at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:38) ~[io.airbyte-airbyte-commons-worker-0.50.30.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:135) ~[io.airbyte-airbyte-workers-0.50.30.jar:?] at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$2(NormalizationActivityImpl.java:175) ~[io.airbyte-airbyte-workers-0.50.30.jar:?] at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.50.30.jar:?] at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.normalize(NormalizationActivityImpl.java:124) ~[io.airbyte-airbyte-workers-0.50.30.jar:?] at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?] 2023-09-25 20:45:09 INFO i.a.w.g.DefaultNormalizationWorker(run):102 - Normalization executed in 0 seconds for job 7. 2023-09-25 20:45:09 normalization > [FATAL tini (7)] exec /airbyte/entrypoint.sh failed: Exec format error

[judiennetSC]

@tomt00001 I believe you should opt for a t3 instance instead of a t2 though. 30% more cost/efficiency ratio :) cf. https://aws.amazon.com/ec2/instance-types/ . Same processor so it should still work exactly the same.

[judiennetSC]

Thanks a lot Tom for testing and succeeding!
Take care

[judiennetSC]

Resolved