time_reopen() does not work for clickhouse destination #747
Description
AxoSyslog
Version of AxoSyslog
axosyslog 4 (4.14.0.31.g3957fc9.dirty)
Issue
Failure
axosyslog can reconnect to clickhouse server only after a reload
Steps to reproduce
- axosyslog starts with a valid clickhouse-destination() config also added a time-reopen(1) option
- before axosyslog starts clickhouse server does not operates
- start clickhouse-sever after axosyslog started
- notice that axosyslog can not connect to clickhouse server
Configuration
@version: 4.2
source source_26331761693986844205907119670118529009 {
example_msg_generator (
num(1)
template("test message db8bd6a0-d992-42d5-8169-b1909f08621d")
);
};
destination destination_208964940347454292653653180649464498565 {
clickhouse (
database(default)
table(test_table)
user(default)
password("password")
schema("message" "String" => "$MSG")
time_reopen(1)
);
};
log {
source(source_26331761693986844205907119670118529009);
destination(destination_208964940347454292653653180649464498565);
};
Input and output logs (if possible)
[2025-08-01T07:47:29.693864] Incoming generated message; msg='test message db8bd6a0-d992-42d5-8169-b1909f08621d'
[2025-08-01T07:47:29.693864] >>>>>> Source side message processing begin; location='syslog_ng_server.conf:4:5', msg='0x60e520ef0c80', rcptid='1'
[2025-08-01T07:47:29.693864] Setting value; name='HOST_FROM', value='micek-ThinkPad-T14-Gen-4', type='string', msg='0x60e520ef0c80', rcptid='1'
[2025-08-01T07:47:29.693864] Setting value; name='HOST', value='micek-ThinkPad-T14-Gen-4', type='string', msg='0x60e520ef0c80', rcptid='1'
[2025-08-01T07:47:29.693864] Setting tag; name='.source.source_26331761693986844205907119670118529009', value='1', msg='0x60e520ef0c80'
[2025-08-01T07:47:29.693864] Setting value; name='SOURCE', value='source_26331761693986844205907119670118529009', type='string', msg='0x60e520ef0c80', rcptid='1'
[2025-08-01T07:47:29.693864] Window size adjustment; old_window_size='99', window_size_increment='1', suspended_before_increment='FALSE', last_ack_type_is_suspended='FALSE'
[2025-08-01T07:47:29.693864] <<<<<< Source side message processing finish; location='syslog_ng_server.conf:4:5', msg='0x60e520ef0c80', rcptid='1'
[2025-08-01T07:47:29.753146] Message(s) available in queue, starting inserts; driver='destination_208964940347454292653653180649464498565#0', worker_index='0'
[2025-08-01T07:47:29.753146] Message added to ClickHouse batch; location='syslog_ng_server.conf:11:5'
[2025-08-01T07:47:29.753146] Flushing batch; driver='destination_208964940347454292653653180649464498565#0', worker_index='0', batch_size='1'
[2025-08-01T07:47:29.753146] ClickHouse server responded with a temporary error status code, retrying after time-reopen() seconds; error_code='14', error_message='failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9100: Failed to connect to remote host: Connection refused', error_details=''
[2025-08-01T07:47:29.753146] Server disconnected while preparing messages for sending, trying again; driver='destination_208964940347454292653653180649464498565#0', location='syslog_ng_server.conf:11:5', worker_index='0', time_reopen='1', batch_size='1'
[2025-08-01T07:47:29.759500] Control command thread has started; control_command='STATS CSV'
[2025-08-01T07:47:29.759500] Control command thread is exiting now; control_command='STATS CSV'
[2025-08-01T07:47:29.760840] EOF on control channel, closing connection;
[2025-08-01T07:47:29.787941] Control command thread has started; control_command='STATS CSV'
[2025-08-01T07:47:29.787941] Control command thread is exiting now; control_command='STATS CSV'
[2025-08-01T07:47:29.789324] EOF on control channel, closing connection;
[2025-08-01T07:47:29.795416] Control command thread has started; control_command='STATS CSV'
[2025-08-01T07:47:29.795416] Control command thread is exiting now; control_command='STATS CSV'
[2025-08-01T07:47:29.796636] EOF on control channel, closing connection;
[2025-08-01T07:47:29.802789] Control command thread has started; control_command='STATS CSV'
[2025-08-01T07:47:29.802789] Control command thread is exiting now; control_command='STATS CSV'
[2025-08-01T07:47:29.804309] EOF on control channel, closing connection;
[2025-08-01T07:47:30.228131] Control command thread has started; control_command='STATS CSV'
[2025-08-01T07:47:30.228131] Control command thread is exiting now; control_command='STATS CSV'
[2025-08-01T07:47:30.230200] EOF on control channel, closing connection;
[2025-08-01T07:47:30.756984] Message(s) available in queue, starting inserts; driver='destination_208964940347454292653653180649464498565#0', worker_index='0'
[2025-08-01T07:47:30.756984] Message added to ClickHouse batch; location='syslog_ng_server.conf:11:5'
[2025-08-01T07:47:30.756984] Flushing batch; driver='destination_208964940347454292653653180649464498565#0', worker_index='0', batch_size='1'
[2025-08-01T07:47:30.756984] ClickHouse server responded with a temporary error status code, retrying after time-reopen() seconds; error_code='14', error_message='failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9100: Failed to connect to remote host: Connection refused', error_details=''
[2025-08-01T07:47:30.756984] Server disconnected while preparing messages for sending, trying again; driver='destination_208964940347454292653653180649464498565#0', location='syslog_ng_server.conf:11:5', worker_index='0', time_reopen='1', batch_size='1'
[2025-08-01T07:47:31.758476] Message(s) available in queue, starting inserts; driver='destination_208964940347454292653653180649464498565#0', worker_index='0'
[2025-08-01T07:47:31.758476] Message added to ClickHouse batch; location='syslog_ng_server.conf:11:5'
[2025-08-01T07:47:31.758476] Flushing batch; driver='destination_208964940347454292653653180649464498565#0', worker_index='0', batch_size='1'
[2025-08-01T07:47:31.758476] ClickHouse server responded with a temporary error status code, retrying after time-reopen() seconds; error_code='14', error_message='failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9100: Failed to connect to remote host: Connection refused', error_details=''
[2025-08-01T07:47:31.758476] Server disconnected while preparing messages for sending, trying again; driver='destination_208964940347454292653653180649464498565#0', location='syslog_ng_server.conf:11:5', worker_index='0', time_reopen='1', batch_size='1'