Race-condition / use-after-free on http console endpoints.

[w1ll-i-code]

Describe the bug

The endpoints /v1/console/execute-script and /v1/console/auto-complete-script don't take the mutex when inserting or reading session state from the map l_ApiScriptFrames resulting in a seg fault and a core-dump if the tree re-balances during the cleanup in ScriptFrameCleanupHandler

To Reproduce

#!/bin/bash

ICINGA_HOST="localhost"
ICINGA_PORT="5665"
ICINGA_USER="root"
ICINGA_PASS="a758a9fdfc8cd286"

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

echo -e "${GREEN}=== RAPID-FIRE ===${NC}"
echo ""

# Crash detection in background
monitor_crash() {
  while true; do
    if dmesg -T | tail -30 | grep -q "icinga2.*segfault"; then
      echo -e "\n${RED}!!! SEGFAULT DETECTED !!!${NC}"
      dmesg -T | tail -50 | grep -A10 "icinga2"
      journalctl -u icinga2-master --since "1 minute ago" | tail -100
      pkill -P $$
      exit 0
    fi
    sleep 0.5
  done
}

monitor_crash &
MONITOR_PID=$!
trap "kill $MONITOR_PID 2>/dev/null; pkill -P $$" EXIT INT TERM

# Function for rapid burst
rapid_burst() {
  local burst_id=$1
  local session_prefix="rapid-$burst_id"

  # Launch requests as fast as possible
  for i in {1..500}; do
    curl -k -s -u "$ICINGA_USER:$ICINGA_PASS" \
      -H 'Accept: application/json' \
      -X POST "https://$ICINGA_HOST:$ICINGA_PORT/v1/console/execute-script" \
      -d "{\"session\": \"$session_prefix-$i\", \"command\": \"1+1\"}" \
      > /dev/null 2>&1 &
  done
}

echo -e "${YELLOW}Starting 20 rapid-fire cycles...${NC}"
echo "Each cycle: 500 requests, abort after 3 seconds, restart"
echo ""

for cycle in {1..20}; do
  LOAD=$(uptime | awk -F'load average:' '{print $2}' | awk '{print $1}' | tr -d ',')
  THREADS=$(ps -eLf | grep icinga2 | grep -v grep | wc -l)

  echo -e "${GREEN}Cycle $cycle/20${NC} - Load: ${YELLOW}$LOAD${NC}, Threads: $THREADS"

  # Launch burst
  rapid_burst $cycle

  # Critical: Let it run
  sleep 3

  # Kill all curl processes (simulates abort)
  pkill -9 curl 2>/dev/null

  # Check for crash
  if [ -f /var/lib/cores/core.icinga2.* ]; then
    echo -e "${RED}!!! CORE DUMP FOUND !!!${NC}"
    ls -lh /var/lib/cores/core.icinga2.*
    exit 0
  fi

  # NO DELAY - restart immediately (key to building up load)

  # Every 5 cycles, show status
  if [ $((cycle % 5)) -eq 0 ]; then
    echo "  Current load: $LOAD"
    if (( $(echo "$LOAD > 10" | bc -l) )); then
      echo -e "  ${YELLOW}Load elevated - continuing pressure${NC}"
    fi
  fi
done

echo ""
echo -e "${GREEN}Phase 1 complete. Load built up to:${NC}"
uptime

echo ""
echo -e "${YELLOW}Phase 2: Sustain high load for 60 seconds${NC}"
echo "This keeps map under constant pressure while cleanup timer runs..."

# Sustain the pressure
END_TIME=$(($(date +%s) + 60))
BURST_ID=1000

while [ $(date +%s) -lt $END_TIME ]; do
  rapid_burst $BURST_ID
  BURST_ID=$((BURST_ID + 1))
  sleep 0.5  # Brief pause between bursts

  # Kill every 3 seconds to prevent overwhelming
  if [ $((BURST_ID % 6)) -eq 0 ]; then
    pkill -9 curl 2>/dev/null
  fi
done

pkill -9 curl 2>/dev/null
wait

echo ""
echo -e "${GREEN}Test complete${NC}"
echo "Final load: $(uptime | awk -F'load average:' '{print $2}')"
echo ""
echo "Check for crashes:"
echo "  dmesg -T | grep icinga2"
echo "  ls -lh /var/lib/cores/"

Expected behavior

I expect the application not to crash.

Screenshots

(gdb) where
#0  0x00007fcdd666f792 in std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) () from /lib64/libstdc++.so.6
#1  0x0000000000b57074 in std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::ApiScriptFrame>, std::_Select1st<std::pair<icinga::String const, icinga::ApiScriptFrame> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::ApiScriptFrame> > >::_M_erase_aux (    __position=..., this=0x1589b40 <l_ApiScriptFrames>) at /usr/include/c++/8/bits/stl_tree.h:2493
#2  std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::ApiScriptFrame>, std::_Select1st<std::pair<icinga::String const, icinga::ApiScriptFrame> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::ApiScriptFrame> > >::_M_erase_aux (__last=      {first = {static NPos = 18446744073709551615, m_Data = "51348ff5-0fde-4d17-b1e5-0d12ed071680"}, second = {Seen = 1764943640.7009151, NextLine = 2, Lines = std::map with 1 element = {[{static NPos = 18446744073709551615, m_Data = "<1>"}] = {static NPos = 18446744073709551615, m_Data = "existing = get_service(\"some-sercice\",\"Elastic Ingest Status - system.system (elastic_agent)\")\nf = function() {\n\nobject Service \"Elastic Ingest Status - system.system (elastic_agent)\" {\n    "...}}, Locals = {px = 0x7fcc941549e0}}}, __first=      {first = {static NPos = 18446744073709551615, m_Data = "51348ff5-0fde-4d17-b1e5-0d12ed071680"}, second = {Seen = 1764943640.7009151, NextLine = 2, Lines = std::map with 1 element = {[{static NPos = 18446744073709551615, m_Data = "<1>"}] = {static NPos = 18446744073709551615, m_Data = "existing = get_service(\"some-service\",\"Elastic Ingest Status - system.system (elastic_agent)\")\nf = function() {\n\nobject Service \"Elastic Ingest Status - system.system (elastic_agent)\" {\n    "...}}, Locals = {px = 0x7fcc941549e0}}}, this=0x1589b40 <l_ApiScriptFrames>) at /usr/include/c++/8/bits/stl_tree.h:2514
#3  std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::ApiScriptFrame>, std::_Select1st<std::pair<icinga::String const, icinga::ApiScriptFrame> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::ApiScriptFrame> > >::erase (__x=...,     this=0x1589b40 <l_ApiScriptFrames>) at /usr/include/c++/8/bits/stl_tree.h:2525
#4  std::map<icinga::String, icinga::ApiScriptFrame, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::ApiScriptFrame> > >::erase (__x=..., this=0x1589b40 <l_ApiScriptFrames>) at /usr/include/c++/8/bits/stl_map.h:1068
#5  ScriptFrameCleanupHandler () at /usr/src/debug/icinga2-2.15.1_neteye1.64.0-1.el8.x86_64/lib/remote/consolehandler.cpp:42
#6  <lambda()>::<lambda(const icinga::Timer* const&)>::operator() (__closure=<optimized out>)    at /usr/src/debug/icinga2-2.15.1_neteye1.64.0-1.el8.x86_64/lib/remote/consolehandler.cpp:51
#7  boost::detail::function::void_function_obj_invoker1<EnsureFrameCleanupTimer()::<lambda()>::<lambda(const icinga::Timer* const&)>, void, const icinga::Timer* const&>::invoke(boost::detail::function::function_buffer &, const icinga::Timer * const&) (function_obj_ptr=..., a0=<optimized out>)    at /usr/include/boost/function/function_template.hpp:159
#8  0x0000000000aebd19 in boost::signals2::detail::signal_impl<void (icinga::Timer const* const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (icinga::Timer const* const&)>, boost::function<void (boost::signals2::connection const&, icinga::Timer const* const&)>, boost::signals2::mutex>::operator()(icinga::Timer const* const&) (this=0x7fcc44166460, args#0=<optimized out>) at /usr/include/boost/function/function_template.hpp:673
#9  0x0000000000a6448c in icinga::Timer::Call() () at /usr/include/boost/signals2/detail/signal_template.hpp:720
#10 0x0000000000ac5fe1 in icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}::operator()() const    () at /usr/include/c++/8/bits/std_function.h:682

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.15.1-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Red Hat Enterprise Linux
  Platform version: 8.10 (Ootpa)
  Kernel: Linux
  Kernel version: 6.17.10-300.fc43.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.5.0
  Build host: neteye4-dev
  OpenSSL version: OpenSSL 1.1.1k  FIPS 25 Mar 2021

Application information:

General paths:
  Config directory: /neteye/local/icinga2/conf/icinga2
  Data directory: /neteye/local/icinga2/data/lib/icinga2
  Log directory: /neteye/local/icinga2/data/log/icinga2
  Cache directory: /neteye/local/icinga2/data/cache/icinga2
  Spool directory: /neteye/local/icinga2/data/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /neteye/local/icinga2/conf
  Run directory (base): /run
  Local state directory: /neteye/local/icinga2/data

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /neteye/local/icinga2/data/lib/icinga2/icinga2.state
  Modified attributes path: /neteye/local/icinga2/data/lib/icinga2/modified-attributes.conf
  Objects path: /neteye/local/icinga2/data/cache/icinga2/icinga2.debug
  Vars path: /neteye/local/icinga2/data/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

• Operating System and version: Red Hat Enterprise Linux release 8.10 (Ootpa)
• Enabled features (icinga2 feature list): api checker debuglog icingadb ido-mysql influxdb livestatus mainlog neteye_datastreamwriter notification