AWS Lambda Executor - External Executor Id Column Length Issue

[chsanjeev]

Apache Airflow version

3.1.5

If "Other Airflow 3 version" selected, which one?

No response

What happened?

I am testing the AWS Lambda Executor (I am aware that it is still in experimental mode), and noticed the scheduler is crashing when the task external executor id chars are more than 250 chars length.

This applies to any executor's where tasks are externally managed with long text.

Root Cause: external_executor_id column length is currently set to 250 in the tables task_instance and task_instance_history. Quick fix is to increase the length of the column external_executor_id, but not sure if it is ideal.

What you think should happen instead?

Crashing scheduler with following message:

self.dialect.do_execute(
File "/usr/local/lib64/python3.12/site-packages/sqlalchemy/engine/default.py", Line 736, in do_execute cursor-execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(250)

[SQL: UPDATE task_instance SET updated_at=%(updated_at)s, external_executor_1d=%(external_executor_id)s WHERE task_instance.1d = 8(task_instance_1d) s]
Iparameters: ('updated_at': datetime.datetime(2025,12, 24,23, 31, 50, 117583, tzinfo=Timezone('UTC')), 'external_ executor_id": 'f"dag_id": "ZZ_YY_ZZZZZZZZZ_YYYYYYY", "task_id":
ааааааа.bbbbbbbbbbbbbbbbbbb","run_id":"kbbkkbpkk2 2025_1__.sv. bz2-loadd_n-2025.12.24_23.30.10",
"try_number"; 1, "map_index": -1)', 'task_instance_id': UUID('019b52b2-d84b-7fd4-aa9f-f55305792255')}] (Background on this error at: https://sqlalche.me/e/14/9h9hl)

How to reproduce

Run DAG making aws lambda executor as the primary executor and keep long dag id, task id and run_id

Operating System

rhel 9

Versions of Apache Airflow Providers

Airflow 3.1.5
apache-airflow-providers-amazon 9.18.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[suii2210]

Thanks for the detailed report -this does look like a genuine schema-level bug.

The external_executor_id column is currently limited to VARCHAR(250) in
task_instance and task_instance_history, but executors like the AWS Lambda
Executor can legitimately generate longer, structured identifiers (dag_id,
task_id, run_id, map_index, etc.).

Since external_executor_id is opaque metadata from Airflow’s perspective,
enforcing such a small fixed length can cause scheduler crashes, as seen here.
This issue is also not AWS-specific and could affect any external executor that
manages tasks outside Airflow.

A more robust fix would be to increase the column type (e.g. to TEXT) for both
tables, along with a migration, rather than relying on executors to truncate
IDs.
Happy to contribute in this .

[Arunodoy18]

Hi, this too looks intresting, I would like to take on this issue .
Thank you

[jroachgolf84]

@Arunodoy18, please feel free to open up a PR for this issue.