Airflow Scheduler Getting fail

[taha-alphavu]

Apache Airflow version

Other Airflow 1.10.15 version (please specify below)

What happened

Airflow Scheduler is getting failed randomly and after checking the server I have got this issue. We had Reset and initialize the Database but still same error. Can anyone help us out in this error?

[2023-05-05 12:23:18,597] {__init__.py:56} WARNING - Failed operation _get_task_meta_for.  Retrying 2 more times.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ProtocolViolation: invalid string in message


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 51, in _inner
    return fun(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 153, in _get_task_meta_for
    task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__
    return self._execute_and_instances(context)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/home/ubuntu/anaconda3/envs/alphavu/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.errors.ProtocolViolation) invalid string in message

[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback 
FROM celery_taskmeta 
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': '2652a5ec-f13f-455b-9aab-0e98698b75c6'}]
(Background on this error at: http://sqlalche.me/e/4xp6)

What you think should happen instead

No response

How to reproduce

We had tried resolving it through resetting and initializing database but still could not found anything.

Operating System

Ubuntu 18.04

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

As discussed before - this is a very specific error you have likely due to some mismatch between the version of client and server or encoding in your client and server, you need to look there.

I converted it into a discussion as this is not a common problem with airflow but some specific issue you have with your Postgres client server.

If you want some help here, I think it would be best if you gather all the information about anything special and non-standard you have in your setup. My guess is that you are using some non-standard client or non-standard DB server that generates the issue, or some strange encoding setting on oither of those.

[taha-alphavu]

We are using Postgres version 11 and it is deployed on AWS RDS.

[potiuk]

BTW. If you have Airlfow 1.10.15 (that's an interesting one) then you should look at airflow 1.10.15 prerequisites and see if your postgres version matches them. There is a possibility you have mismatch there.

[taha-alphavu]

Previously like on Monday it was working fine. But suddenly it got failed and I got this error.

[taha-alphavu]

Is there any way to debug Airflow Scheduler command to check Which values are creating problem in Database? Our server was first stopped due to gunicorn time out error and we resolved and then scheduler started getting failed

[potiuk]

Then this is somethign on AWS RDS side if you have not changed anything on Airflow. There is no way it can be on airflow side if it started to happen suddenly. Open an issue with AWS RDS.

It's very likely they stopped supporting the client and library versions that you have installed in your machine. Having 3 years past end of life software I guess indicates that your OS and libraries are also very old and they simply might have stopped being suppoerted by AWS RDS (I would not be surprised because there likely many security vulnerabilities - for sure there are many fixed in Airflow since 1.10 reached end of life 3 years ago).

Your best option is to upgrede everything including Airflow (preferrably to 2.6.* line. Keeping old softwre not upgraded with even security fixes for years is a recipe for such disasters.

[mubasherjaved2665]

@potiuk even after upgrading both Postgres and Airflow version to latest error is still same. I can see we have model named DagWarning with column name message. So do you think it might be the cause of error, to my guess maybe Airflow trying to push some kind of warning to this model but somehow getting failed (maybe due to message text other than UFT-8 encoding)

[potiuk]

No idea.