filesystem: Fix crash when watch callback returns error or throws (#42554)

agrawroh · web-flow · commit 06e40b2de344 · 2025-12-16T11:35:16.000-05:00
## Description Today, when a filesystem watch callback returns a non-OK status or throws an exception, the error gets propagated to `FileEventImpl` which uses `THROW_IF_NOT_OK`. Since there's no exception handler in the `libevent` loop, this causes `std::terminate` to be called, which crashes Envoy. **Stack Trace:** ``` Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.119][234999][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.core.v3.HeaderValueOption Using deprecated option 'envoy.config.core.v3.HeaderValueOption.append' from file base.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.120][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '0_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.123][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '1_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.126][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '2_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.127][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '3_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.128][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '4_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.130][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '5_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.132][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '6_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.134][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_untrusted_regional_transparent_tunnel_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.135][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_app_trusted_regional_transparent_tunnel_listener' Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][main] [source/exe/terminate_handler.cc:36] std::terminate called! Uncaught unknown exception, see trace. Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers): Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.100][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x558d8da5784f] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.102][234999][critical][backtrace] [./source/server/backtrace.h:123] #1: [0x558d8edd8673] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.104][234999][critical][backtrace] [./source/server/backtrace.h:123] #2: [0x558d8e3b120b] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.106][234999][critical][backtrace] [./source/server/backtrace.h:121] #3: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.108][234999][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x558d8e3998d2] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.109][234999][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x558d8e393de6] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.111][234999][critical][backtrace] [./source/server/backtrace.h:121] #6: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.113][234999][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x558d8e710823] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.115][234999][critical][backtrace] [./source/server/backtrace.h:121] #8: event_base_loop [0x558d8e70d4a1] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.117][234999][critical][backtrace] [./source/server/backtrace.h:121] #9: Envoy::Server::InstanceBase::run() [0x558d8daa2b99] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.119][234999][critical][backtrace] [./source/server/backtrace.h:121] #10: Envoy::MainCommonBase::run() [0x558d8da4327a] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.121][234999][critical][backtrace] [./source/server/backtrace.h:121] #11: Envoy::MainCommon::main() [0x558d8da44234] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] #12: main [0x558d8afcc11c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:123] #13: [0x7f1d54073efb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] #14: __libc_start_main [0x7f1d54073fbb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #15: _start [0x558d8afcc02e] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:129] Caught Aborted, suspect faulting address 0x395f7 Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers): Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7f1d54089c90] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #1: gsignal [0x7f1d54089bde] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #2: abort [0x7f1d54072832] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.126][234999][critical][backtrace] [./source/server/backtrace.h:123] #3: [0x558d8da5785c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.128][234999][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x558d8edd8673] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x558d8e3b120b] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:121] #6: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.131][234999][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x558d8e3998d2] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:123] #8: [0x558d8e393de6] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:121] #9: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] #10: [0x558d8e710823] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #11: event_base_loop [0x558d8e70d4a1] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #12: Envoy::Server::InstanceBase::run() [0x558d8daa2b99] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #13: Envoy::MainCommonBase::run() [0x558d8da4327a] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #14: Envoy::MainCommon::main() [0x558d8da44234] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #15: main [0x558d8afcc11c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] #16: [0x7f1d54073efb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #17: __libc_start_main [0x7f1d54073fbb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #18: _start [0x558d8afcc02e] ``` In this change, we are making the `inotify` and `kqueue` watchers handle callback errors gracefully by catching any exceptions using `TRY_ASSERT_MAIN_THREAD`, logging errors instead of propagating them and always returning the `OkStatus` to the event loop. --- **Commit Message:** filesystem: Fix crash when watch callback returns error or throws **Additional Description:** Make `inotify` and `kqueue` watchers handle callback errors gracefully. **Risk Level:** Low **Testing:** CI **Docs Changes:** N/A **Release Notes:** N/A --------- Signed-off-by: Rohit Agrawal <rohit.agrawal@salesforce.com> Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>
diff --git a/source/common/filesystem/BUILD b/source/common/filesystem/BUILD
@@ -120,12 +120,12 @@ envoy_cc_library(
         "//source/common/common:assert_lib",
         "//source/common/common:linked_object",
         "//source/common/common:minimal_logger_lib",
+        "//source/common/common:thread_lib",
         "//source/common/common:utility_lib",
         "//source/common/network:default_socket_interface_lib",
     ] + select({
         "//bazel:windows_x86_64": [
             "//source/common/api:os_sys_calls_lib",
-            "//source/common/common:thread_lib",
         ],
         "//conditions:default": [],
     }),
diff --git a/source/common/filesystem/inotify/watcher_impl.cc b/source/common/filesystem/inotify/watcher_impl.cc
@@ -10,6 +10,7 @@
 
 #include "source/common/common/assert.h"
 #include "source/common/common/fmt.h"
+#include "source/common/common/thread.h"
 #include "source/common/common/utility.h"
 #include "source/common/filesystem/watcher_impl.h"
 
@@ -52,6 +53,28 @@ absl::Status WatcherImpl::addWatch(absl::string_view path, uint32_t events, OnCh
   return absl::OkStatus();
 }
 
+void WatcherImpl::callAndLogOnError(OnChangedCb& cb, uint32_t events, const std::string& file) {
+  TRY_ASSERT_MAIN_THREAD {
+    const absl::Status status = cb(events);
+    if (!status.ok()) {
+      // Use ENVOY_LOG_EVERY_POW_2 to avoid log spam if a callback keeps failing.
+      ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' returned error: {}", file,
+                            status.message());
+    }
+  }
+  END_TRY
+  MULTI_CATCH(
+      const std::exception& e,
+      {
+        ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' threw exception: {}", file,
+                              e.what());
+      },
+      {
+        ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' threw unknown exception",
+                              file);
+      });
+}
+
 absl::Status WatcherImpl::onInotifyEvent() {
   while (true) {
     // The buffer needs to be suitably aligned to store the first inotify_event structure.
@@ -90,10 +113,10 @@ absl::Status WatcherImpl::onInotifyEvent() {
         if (watch.events_ & events) {
           if (watch.file_ == file) {
             ENVOY_LOG(debug, "matched callback: file: {}", file);
-            RETURN_IF_NOT_OK(watch.cb_(events));
+            callAndLogOnError(watch.cb_, events, file);
           } else if (watch.file_.empty()) {
             ENVOY_LOG(debug, "matched callback: directory: {}", file);
-            RETURN_IF_NOT_OK(watch.cb_(events));
+            callAndLogOnError(watch.cb_, events, file);
           }
         }
       }
diff --git a/source/common/filesystem/inotify/watcher_impl.h b/source/common/filesystem/inotify/watcher_impl.h
@@ -40,6 +40,7 @@ class WatcherImpl : public Watcher, Logger::Loggable<Logger::Id::file> {
   };
 
   absl::Status onInotifyEvent();
+  void callAndLogOnError(OnChangedCb& cb, uint32_t events, const std::string& file);
 
   Filesystem::Instance& file_system_;
   int inotify_fd_;
diff --git a/source/common/filesystem/kqueue/watcher_impl.cc b/source/common/filesystem/kqueue/watcher_impl.cc
@@ -8,6 +8,7 @@
 
 #include "source/common/common/assert.h"
 #include "source/common/common/fmt.h"
+#include "source/common/common/thread.h"
 #include "source/common/common/utility.h"
 #include "source/common/filesystem/watcher_impl.h"
 
@@ -116,20 +117,31 @@ absl::Status WatcherImpl::onKqueueEvent() {
 
     absl::StatusOr<PathSplitResult> pathname_or_error =
         file_system_.splitPathFromFilename(file->file_);
-    RETURN_IF_NOT_OK_REF(pathname_or_error.status());
+    if (!pathname_or_error.ok()) {
+      // Path split failure is permanent and we can't recover.
+      // We remove the broken watch to avoid repeated failures.
+      ENVOY_LOG(warn, "Failed to split path '{}', removing watch: {}", file->file_,
+                pathname_or_error.status().message());
+      removeWatch(file);
+      continue;
+    }
     PathSplitResult& pathname = pathname_or_error.value();
 
     if (file->watching_dir_) {
       if (event.fflags & NOTE_DELETE) {
-        // directory was deleted
+        // Directory was deleted.
         removeWatch(file);
         return absl::OkStatus();
       }
 
       if (event.fflags & NOTE_WRITE) {
-        // directory was written -- check if the file we're actually watching appeared
+        // Directory was written -- check if the file we're actually watching appeared.
         auto file_or_error = addWatch(file->file_, file->events_, file->callback_, true);
-        RETURN_IF_NOT_OK_REF(file_or_error.status());
+        if (!file_or_error.ok()) {
+          ENVOY_LOG_EVERY_POW_2(warn, "Failed to re-add watch for '{}': {}", file->file_,
+                                file_or_error.status().message());
+          continue;
+        }
         FileWatchPtr new_file = file_or_error.value();
         if (new_file != nullptr) {
           removeWatch(file);
@@ -150,7 +162,11 @@ absl::Status WatcherImpl::onKqueueEvent() {
         removeWatch(file);
 
         auto file_or_error = addWatch(file->file_, file->events_, file->callback_, true);
-        RETURN_IF_NOT_OK_REF(file_or_error.status());
+        if (!file_or_error.ok()) {
+          ENVOY_LOG_EVERY_POW_2(warn, "Failed to re-add watch for '{}': {}", file->file_,
+                                file_or_error.status().message());
+          continue;
+        }
         FileWatchPtr new_file = file_or_error.value();
         if (new_file == nullptr) {
           return absl::OkStatus();
@@ -173,11 +189,34 @@ absl::Status WatcherImpl::onKqueueEvent() {
 
     if (events & file->events_) {
       ENVOY_LOG(debug, "matched callback: file: {}", file->file_);
-      RETURN_IF_NOT_OK(file->callback_(events));
+      callAndLogOnError(file->callback_, events, file->file_);
     }
   }
   return absl::OkStatus();
 }
 
+void WatcherImpl::callAndLogOnError(Watcher::OnChangedCb& cb, uint32_t events,
+                                    const std::string& file) {
+  TRY_ASSERT_MAIN_THREAD {
+    const absl::Status status = cb(events);
+    if (!status.ok()) {
+      // Use ENVOY_LOG_EVERY_POW_2 to avoid log spam if a callback keeps failing.
+      ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' returned error: {}", file,
+                            status.message());
+    }
+  }
+  END_TRY
+  MULTI_CATCH(
+      const std::exception& e,
+      {
+        ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' threw exception: {}", file,
+                              e.what());
+      },
+      {
+        ENVOY_LOG_EVERY_POW_2(warn, "Filesystem watch callback for '{}' threw unknown exception",
+                              file);
+      });
+}
+
 } // namespace Filesystem
 } // namespace Envoy
diff --git a/source/common/filesystem/kqueue/watcher_impl.h b/source/common/filesystem/kqueue/watcher_impl.h
@@ -46,6 +46,7 @@ class WatcherImpl : public Watcher, Logger::Loggable<Logger::Id::file> {
   absl::StatusOr<FileWatchPtr> addWatch(absl::string_view path, uint32_t events,
                                         Watcher::OnChangedCb cb, bool pathMustExist);
   void removeWatch(FileWatchPtr& watch);
+  void callAndLogOnError(OnChangedCb& cb, uint32_t events, const std::string& file);
 
   Filesystem::Instance& file_system_;
   int queue_;
diff --git a/test/common/filesystem/BUILD b/test/common/filesystem/BUILD
@@ -34,10 +34,12 @@ envoy_cc_test(
     srcs = ["watcher_impl_test.cc"],
     rbe_pool = "6gig",
     deps = [
+        "//envoy/common:exception_lib",
         "//source/common/common:assert_lib",
         "//source/common/event:dispatcher_includes",
         "//source/common/event:dispatcher_lib",
         "//source/common/filesystem:watcher_lib",
         "//test/test_common:environment_lib",
+        "//test/test_common:logging_lib",
     ],
 )
diff --git a/test/common/filesystem/watcher_impl_test.cc b/test/common/filesystem/watcher_impl_test.cc
@@ -1,11 +1,14 @@
 #include <cstdint>
 #include <fstream>
 
+#include "envoy/common/exception.h"
+
 #include "source/common/common/assert.h"
 #include "source/common/event/dispatcher_impl.h"
 #include "source/common/filesystem/watcher_impl.h"
 
 #include "test/test_common/environment.h"
+#include "test/test_common/logging.h"
 #include "test/test_common/utility.h"
 
 #include "gmock/gmock.h"
@@ -214,5 +217,92 @@ TEST_F(WatcherImplTest, SymlinkAtomicRename) {
 }
 #endif
 
+// Test that callback returning error status is logged and doesn't crash.
+TEST_F(WatcherImplTest, CallbackReturnsErrorStatus) {
+  Filesystem::WatcherPtr watcher = dispatcher_->createFilesystemWatcher();
+
+  TestEnvironment::createPath(TestEnvironment::temporaryPath("envoy_test"));
+  std::ofstream file(TestEnvironment::temporaryPath("envoy_test/watcher_target"));
+
+  WatchCallback callback;
+  EXPECT_CALL(callback, called(Watcher::Events::Modified));
+  ASSERT_TRUE(watcher
+                  ->addWatch(TestEnvironment::temporaryPath("envoy_test/watcher_target"),
+                             Watcher::Events::Modified,
+                             [&](uint32_t events) {
+                               callback.called(events);
+                               dispatcher_->exit();
+                               // Return an error status - should be logged but not crash.
+                               return absl::InternalError("simulated callback error");
+                             })
+                  .ok());
+  dispatcher_->run(Event::Dispatcher::RunType::NonBlock);
+
+  EXPECT_LOG_CONTAINS("warn", "Filesystem watch callback for", file << "text" << std::flush;
+                      file.close(); dispatcher_->run(Event::Dispatcher::RunType::Block););
+}
+
+// Test that callback throwing exception is caught and logged.
+TEST_F(WatcherImplTest, CallbackThrowsException) {
+  Filesystem::WatcherPtr watcher = dispatcher_->createFilesystemWatcher();
+
+  TestEnvironment::createPath(TestEnvironment::temporaryPath("envoy_test"));
+  std::ofstream file(TestEnvironment::temporaryPath("envoy_test/watcher_target"));
+
+  WatchCallback callback;
+  EXPECT_CALL(callback, called(Watcher::Events::Modified));
+  ASSERT_TRUE(watcher
+                  ->addWatch(TestEnvironment::temporaryPath("envoy_test/watcher_target"),
+                             Watcher::Events::Modified,
+                             [&](uint32_t events) -> absl::Status {
+                               callback.called(events);
+                               dispatcher_->exit();
+                               // Throw an exception - should be caught and logged.
+                               throw EnvoyException("simulated callback exception");
+                             })
+                  .ok());
+  dispatcher_->run(Event::Dispatcher::RunType::NonBlock);
+
+  EXPECT_LOG_CONTAINS("warn", "threw exception", file << "text" << std::flush; file.close();
+                      dispatcher_->run(Event::Dispatcher::RunType::Block););
+}
+
+// Test that multiple callbacks can fail without affecting each other.
+TEST_F(WatcherImplTest, MultipleCallbacksWithErrors) {
+  Filesystem::WatcherPtr watcher = dispatcher_->createFilesystemWatcher();
+
+  TestEnvironment::createPath(TestEnvironment::temporaryPath("envoy_test"));
+  std::ofstream file(TestEnvironment::temporaryPath("envoy_test/watcher_target"));
+
+  int callback_count = 0;
+  ASSERT_TRUE(watcher
+                  ->addWatch(TestEnvironment::temporaryPath("envoy_test/watcher_target"),
+                             Watcher::Events::Modified,
+                             [&](uint32_t) {
+                               callback_count++;
+                               if (callback_count >= 2) {
+                                 dispatcher_->exit();
+                               }
+                               // First callback returns error, second returns OK.
+                               if (callback_count == 1) {
+                                 return absl::InternalError("first callback error");
+                               }
+                               return absl::OkStatus();
+                             })
+                  .ok());
+  dispatcher_->run(Event::Dispatcher::RunType::NonBlock);
+
+  // Trigger first modification. The first callback returns error, but watcher continues.
+  file << "text1" << std::flush;
+  dispatcher_->run(Event::Dispatcher::RunType::NonBlock);
+
+  // Trigger second modification. It should still work.
+  file << "text2" << std::flush;
+  file.close();
+  dispatcher_->run(Event::Dispatcher::RunType::Block);
+
+  EXPECT_EQ(2, callback_count);
+}
+
 } // namespace Filesystem
 } // namespace Envoy
