Settings for retrying failed Apprise notifications?

[ifmvaiea]

Hello,

After seeing approximately 33-50% of my Telegram notifications fail while running the scripts, are there any settings to retry the Apprise notifications upon failure instead of doing nothing?

Thank you!

[cyberjunky]

I use apprise in all of my scripts, never heard or seen this issue, there must be something wrong with your setup.. or type of apprise notification is not stable.

[ifmvaiea]

How would I begin to troubleshoot this issue? I'm using Windows 10 Python 3.9 and, for the most part, the other functions work as expected. By type of Apprise notification not being stable, I'm using Telegram currently. Is that what you are referring to? On the next run I'm sure I'll have some error messages to share if that would help.

[cyberjunky]

Telegram is the most used type so if the telegram channel is unique and only used by the script this shall not be the issue.. What is important that the scripts push msgs to a queue which get emptied when loop of script is completely run. So watch for any non apprise related errors or delays when script is run, it maybe disturb or delay sending queue.

[ifmvaiea]

Thank you for the prompt reply. My Telegram channel is unique and is only being used by these scripts. Would running MarketCollector and BotUpdater too close to each other cause them to step on themselves? I will look over the errors after the next run in 30 minutes.

[ifmvaiea]

I'm running four of the scripts (MarketCollector, BotUpdater, Compound and DealCluster) and they all have the same "notify-urls".

MarketCollector and BotUpdater just ran and BotUpdater sent a Telegram notification as expected.

MarketCollector reported this for the last 4 lines:

2024-01-12 14:09:15 - marketcollector - DEBUG - Telegram Payload: {'disable_notification': False, 'disable_web_page_preview': True, 'parse_mode': 'HTML', 'text': "[3C Cyber Bot-Helper marketcollector]\r\n \r\nCoinMarketCap; updated 250 coins (1-250) for base 'BTC'.\r\n \r\nCoinGecko; updated 250 coins (1-250) for base 'BTC'.\r\n \r\nCoinMarketCap; updated 250 coins (1-250) for base 'USD'.\r\n \r\nCoinGecko; updated 250 coins (1-250) for base 'USD'.\r\n \r\nBotAssistExplorer; updated for 438 coins the volatility data based on ['coinbase_spot_usd_highest_volatility_day', 'binance_spot_usdt_highest_volatility_day'].\r\n \r\n", 'chat_id': 1649456723}
2024-01-12 14:09:15 - marketcollector - DEBUG - Starting new HTTPS connection (1): api.telegram.org:443
2024-01-12 14:09:15 - marketcollector - WARNING - A connection error occurred sending Telegram:1649456723 notification.
2024-01-12 14:09:15 - marketcollector - DEBUG - Socket Exception: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

[ifmvaiea]

I did a test to see if the problem with MarketCollector and BotUpdater was due to me starting them at the same time and them possibly trying to send messages to Telegram at the same time. I started MarketCollector then waited 15 minutes before starting BotUpdater. As a result each script was trying to send Telegram notifications while the other was resting.

Despite the above changes MarketCollector sent it's Telegram notification just fine but this time BotUpdater made it's changes and didn't send the Telegram notification. BotUpdater had the same error that MarketCollector previously had:

2024-01-12 15:40:45 - botupdater - WARNING - A connection error occurred sending Telegram:1649456723 notification.
2024-01-12 15:40:45 - botupdater - DEBUG - Socket Exception: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

[ifmvaiea]

I have spent the last day searching for the error on Google with not much luck. It looks like the best solution would be to check for the error message and then retry the connection again if needed.

Can you clue me into what file or files control the POST to the Telegram URL? I'm not adverse to digging around and fixing it myself if this is not an issue for others. I can hack existing code until it works using trial and error, but I don't know what file(s) I should be looking to add the check and retry.

Thank you!

[ifmvaiea]

There is another Python Telegram project on github here:

python-telegram-bot/python-telegram-bot#1773

That is having the same issue described as:

[ConnectionResetError - An existing connection was forcibly closed by the remote host]

The author came up with a temporary fix:

I fixed the issue by requesting to the server again. It prints the network error once and then on the 2nd iteration (requesting again), it works. It required me adding a while loop. Here's the code:

while True:
    try:
        resp = self._con_pool.request(*args, **kwargs)
        break
    except urllib3.exceptions.TimeoutError:
        raise TimedOut()
    except urllib3.exceptions.HTTPError as error:
        # HTTPError must come last as its the base urllib3 exception class
        # TODO: do something smart here; for now just raise NetworkError
        print(NetworkError('urllib3 HTTPError {0}'.format(error)))

This is the closest I have come thus far and although the coding would need to be changed it's a start. I still can't find the file to add a loop myself.

Thank you!

[cyberjunky]

Like I said I use the same apprise code in all of my scripts and have never seen this problem myself or heard about it.
This should not be needed, I guess it's due to a faulty string, that the connection gets reset,

Does it look like below, all on one line? (I changed some numbers)

You can see I added the part suggested by apprise when you first run it.

6887777111:BBHf4yYuveogDVcqHdguqvvWSn2CIaWDazw = bot_token
%%40936303224 = chat_id with added %%40 for encoding, (maybe this is missing?)

notify-urls = ["tgram://6887777111:BBHf4yYuveogDVcqHdguqvvWSn2CIaWDazw/%%40936303224/?image=False&detect=yes&silent=no&preview=no&content=before&format=html&overflow=upstream&rto=4.0&cto=4.0&verify=yes"]

[ifmvaiea]

Yes, my Apprise URLs look just like your example. If the URL was wrong all the scripts would fail because all four of the scripts I'm using have the same URL. MarketCollector and BotUpdater take turns failing to send the Telegram messages. I have both running hourly, but offset from each other's start time. Sometimes they both work during the same period, but more often than not one of those two scripts will fail to send to Telegram. Since it is a connection issue I tried turning off AdGuard last night, but the issue persisted.

Thank you for responding to this issue!

[ifmvaiea]

I fixed my issue by editing NotifyTelegram.py. Notification gets sent successfully after first failed try now.

[cyberjunky]

Care to share what you changed?

[amargedon]

Strange issue. I've also multiple scripts running without problems. I think the while loop was added to retry on failure?