Elastalert problem sending alert via googlechat webhook behind proxy

[tientmse62290]

Hi All
After upgraded to 2.3.140 with the latest hotfix I got the problem with alert to googlechat

gg_alert.yml

# (Required)
# Rule name, must be unique
name: "High Severity Event !!!"


# (Required)
# Index to search, wildcard supported
index: "*:so-ids-*"

# (Required)
# Type of alert.
# This rule will monitor a certain field and match if that field changes.
type: frequency

# (Required, change specific)
# The field to look for changes in
num_events: 1
timeframe:
    minutes: 3
buffer_time:
    minutes: 5
allow_buffer_time_overlap: true
filter:
    - query:
       query_string:
        query: "(event.module: suricata AND event.dataset: alert AND event.severity: 3)"

include:
 - rule.name
 - rule.category
 - source.ip
 - source.port
 - destination.ip
 - destination.port
 - event.ingested
 - log.id.uid

alert_text_type: alert_text_only
alert_text: |
  SEVERITY: HIGH
  Detection Rule             : {0}
  Rule Category              : {1}
  Source IP : Port           : {2} : {3}
  Destination IP : Port      : {4} : {5}
  Detection Time______: {6}
  Login to IDS and Check --> https://some.domain/#/hunt?q=log.id.uid%{7}
alert_text_args:
 - rule.name
 - rule.category
 - source.ip
 - source.port
 - destination.ip
 - destination.port
 - event.ingested
 - log.id.uid

# (Required)
# The alert is use when a match is found
alert:
- "googlechat"
googlechat_alert_summary: "Alert"
googlechat_theme_color: '#ff0000'
googlechat_alert_fixed_width: false
googlechat_proxy: proxyhost:8080
googlechat_webhook_url: "https://chat.googleapis.com/v1/spaces/somespaceaddr/messages?key=somekey"

/opt/so/log/elastalert/elastalert.log

2022-07-25 08:48:22,514    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6447dc760>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2386 2022-07-25 08:48:55,922    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6448e73a0>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2387 2022-07-25 08:48:58,695  WARNING apscheduler.scheduler Execution of job "Internal: Handle Pending Alerts (trigger: interval[0:03:00], next run at: 2022-07-25 08:48:58 UTC)" skipped: maximum number of running instances reached (1        )
   2388 2022-07-25 08:48:58,696     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:51:58 UTC)" (scheduled at 2022-07-25 08:48:58.694437+00:00)
   2389 2022-07-25 08:48:58,697     INFO           elastalert Background configuration change check run at 2022-07-25 08:48 UTC
   2390 2022-07-25 08:48:58,698     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:51:58 UTC)" executed successfully
   2391 2022-07-25 08:48:59,878     INFO           elastalert Disabled rules are: []
   2392 2022-07-25 08:48:59,878     INFO           elastalert Sleeping for 179.999546 seconds
   2393 2022-07-25 08:49:44,740  WARNING apscheduler.scheduler Execution of job "Rule: High Severity Event !!! (trigger: interval[0:03:00], next run at: 2022-07-25 08:49:44 UTC)" skipped: maximum number of running instances reached (1)
   2394 2022-07-25 08:50:29,873    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6435406a0>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2395 2022-07-25 08:51:04,177    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6447d1720>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2396 2022-07-25 08:51:58,695  WARNING apscheduler.scheduler Execution of job "Internal: Handle Pending Alerts (trigger: interval[0:03:00], next run at: 2022-07-25 08:51:58 UTC)" skipped: maximum number of running instances reached (1        )
   2397 2022-07-25 08:51:58,696     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:54:58 UTC)" (scheduled at 2022-07-25 08:51:58.694437+00:00)
   2398 2022-07-25 08:51:58,698     INFO           elastalert Background configuration change check run at 2022-07-25 08:51 UTC
   2399 2022-07-25 08:51:58,698     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:54:58 UTC)" executed successfully
   2400 2022-07-25 08:51:59,963     INFO           elastalert Disabled rules are: []
   2401 2022-07-25 08:51:59,963     INFO           elastalert Sleeping for 179.999645 seconds
   2402 2022-07-25 08:52:37,233    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6448e7070>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2403 2022-07-25 08:52:48,596  WARNING apscheduler.scheduler Execution of job "Rule: High Severity Event !!! (trigger: interval[0:03:00], next run at: 2022-07-25 08:52:48 UTC)" skipped: maximum number of running instances reached (1)
   2404 2022-07-25 08:53:12,306    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6435416f0>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2405 2022-07-25 08:54:44,466    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff643541e70>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2406 2022-07-25 08:54:58,695  WARNING apscheduler.scheduler Execution of job "Internal: Handle Pending Alerts (trigger: interval[0:03:00], next run at: 2022-07-25 08:54:58 UTC)" skipped: maximum number of running instances reached (1        )
   2407 2022-07-25 08:54:58,696     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:57:58 UTC)" (scheduled at 2022-07-25 08:54:58.694437+00:00)
   2408 2022-07-25 08:54:58,698     INFO           elastalert Background configuration change check run at 2022-07-25 08:54 UTC
   2409 2022-07-25 08:54:58,698     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 08:57:58 UTC)" executed successfully
   2410 2022-07-25 08:55:00,063     INFO           elastalert Disabled rules are: []
   2411 2022-07-25 08:55:00,063     INFO           elastalert Sleeping for 179.99973 seconds
   2412 2022-07-25 08:55:20,561    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6447de050>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2413 2022-07-25 08:55:51,857  WARNING apscheduler.scheduler Execution of job "Rule: High Severity Event !!! (trigger: interval[0:03:00], next run at: 2022-07-25 08:55:51 UTC)" skipped: maximum number of running instances reached (1)
   2414 2022-07-25 08:56:51,826    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url: /v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff643541d80>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2415 2022-07-25 08:57:28,820    ERROR           elastalert Error while running alert googlechat: Error posting to google chat: HTTPSConnectionPool(host='chat.googleapis.com', port=443): Max retries exceeded with url:/v1/spaces/somespaceaddr/messages?key=somekey (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff643542e90>, 'Connection to chat.g        oogleapis.com timed out. (connect timeout=None)'))
   2416 2022-07-25 08:57:28,856     INFO           elastalert Background alerts thread 0 pending alerts sent at 2022-07-25 08:57 UTC
   2417 2022-07-25 08:57:28,856     INFO apscheduler.executors.default Job "Internal: Handle Pending Alerts (trigger: interval[0:03:00], next run at: 2022-07-25 08:57:58 UTC)" executed successfully
   2418 2022-07-25 08:57:58,696     INFO apscheduler.executors.default Running job "Internal: Handle Pending Alerts (trigger: interval[0:03:00], next run at: 2022-07-25 09:00:58 UTC)" (scheduled at 2022-07-25 08:57:58.694138+00:00)
   2419 2022-07-25 08:57:58,696     INFO apscheduler.executors.default Running job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 09:00:58 UTC)" (scheduled at 2022-07-25 08:57:58.694437+00:00)
   2420 2022-07-25 08:57:58,700     INFO           elastalert Background configuration change check run at 2022-07-25 08:57 UTC
   2421 2022-07-25 08:57:58,700     INFO apscheduler.executors.default Job "Internal: Handle Config Change (trigger: interval[0:03:00], next run at: 2022-07-25 09:00:58 UTC)" executed successfully
   2422 2022-07-25 08:58:00,163     INFO           elastalert Disabled rules are: []
   2423 2022-07-25 08:58:00,163     INFO           elastalert Sleeping for 179.999644 seconds

The proxy is work fine with other servers.
I think I have to hardcode the proxy into script of the elastalert module. My splunk got the same problem with python request and I can't use the proxy input from UI so I hardcode the proxy to script to.

Anyone have same problem and solution?
Thanks

[tientmse62290]

I checked the repo: https://github.com/jertel/elastalert2
They remove the "googlechat_proxy" option so I can't send alert behind a proxy anymore

[dougburks]

Looks like there never was a googlechat_proxy setting based on your discussion on the elastalert2 repo:
jertel/elastalert2#912 (comment)

[tientmse62290]

Hi @dougburks
It's my mistake. In 130 I can send alert to ggchat but after new install to 140 it couldnt send anymore. Logs show that the elastalert could not make connection although the proxy configured at install stage. Do you have any solutions for this?

Thanks & Regards