Skip to content

Conversation

@vogelsgesang
Copy link
Member

@vogelsgesang vogelsgesang commented Oct 31, 2025

Not ready for detailed review, yet. But two high-level questions around ABI stability could already be discussed:

  1. How to make sure we still emit the ~exception_ptr and copy constructor/assignment in the library. The current approach was previously discussed in https://reviews.llvm.org/D122536 (among others by @philnik777).
  2. If we actually want to introduce the new __do_{inc,dec}rement functions or if we want to directly call __cxa_{inc,dec}rement_refcount from the headers

But if we prefer a different approach, I am happy to adjust this PR accordingly


This commit adds an inlined fast-path for nullptrs to the copy constructor, copy assignment and destructor of std::exception_ptr.

Performance results (from libc++'s CI):

Benchmark                               Baseline    Candidate    Difference    % Difference
------------------------------------  ----------  -----------  ------------  --------------
bm_exception_ptr_copy_assign_nonnull       10.00         7.49         -2.51         -25.11%
bm_exception_ptr_copy_assign_null          10.30         0.94         -9.36         -90.91%
bm_exception_ptr_copy_ctor_nonnull          7.52         6.91         -0.61          -8.16%
bm_exception_ptr_copy_ctor_null            11.24         0.62        -10.61         -94.46%
bm_exception_ptr_move_assign_nonnull       14.32         7.20         -7.12         -49.73%
bm_exception_ptr_move_assign_null          10.90         0.94         -9.96         -91.39%
bm_exception_ptr_move_ctor_nonnull         10.95         6.87         -4.08         -37.24%
bm_exception_ptr_move_ctor_null             7.55         0.94         -6.61         -87.57%
bm_exception_ptr_swap_nonnull               0.62         0.62          0.00           0.08%
bm_exception_ptr_swap_null                  7.82         0.94         -6.88         -88.01%

@github-actions
Copy link

github-actions bot commented Oct 31, 2025

✅ With the latest revision this PR passed the C/C++ code formatter.

@vogelsgesang vogelsgesang force-pushed the avogelsgesang-exception-ptr-inline branch from 38269ec to d332adc Compare November 3, 2025 20:34
@vogelsgesang
Copy link
Member Author

vogelsgesang commented Nov 3, 2025

/libcxx-bot benchmark libcxx/test/benchmarks/exception_ptr.bench.cpp

Benchmark results:
Benchmark                               Baseline    Candidate    Difference    % Difference
------------------------------------  ----------  -----------  ------------  --------------
bm_exception_ptr_copy_assign_nonnull       10.00         7.49         -2.51         -25.11%
bm_exception_ptr_copy_assign_null          10.30         0.94         -9.36         -90.91%
bm_exception_ptr_copy_ctor_nonnull          7.52         6.91         -0.61          -8.16%
bm_exception_ptr_copy_ctor_null            11.24         0.62        -10.61         -94.46%
bm_exception_ptr_move_assign_nonnull       14.32         7.20         -7.12         -49.73%
bm_exception_ptr_move_assign_null          10.90         0.94         -9.96         -91.39%
bm_exception_ptr_move_ctor_nonnull         10.95         6.87         -4.08         -37.24%
bm_exception_ptr_move_ctor_null             7.55         0.94         -6.61         -87.57%
bm_exception_ptr_swap_nonnull               0.62         0.62          0.00           0.08%
bm_exception_ptr_swap_null                  7.82         0.94         -6.88         -88.01%
bm_make_exception_ptr/threads:1            33.37        33.82          0.45           1.36%
bm_make_exception_ptr/threads:2            16.68        16.68         -0.00          -0.02%
bm_make_exception_ptr/threads:4             8.35         8.35         -0.00          -0.02%
bm_make_exception_ptr/threads:8             4.19         4.17         -0.02          -0.39%
Geomean                                     8.52         3.26         -5.27         -61.78%

@philnik777
Copy link
Contributor

@vogelsgesang Do you think this is ready for review? If so, please make it ready for review.

@vogelsgesang vogelsgesang marked this pull request as ready for review November 4, 2025 10:44
@vogelsgesang vogelsgesang requested a review from a team as a code owner November 4, 2025 10:44
@llvmbot llvmbot added the libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi. label Nov 4, 2025
@llvmbot
Copy link
Member

llvmbot commented Nov 4, 2025

@llvm/pr-subscribers-libcxx

Author: Adrian Vogelsgesang (vogelsgesang)

Changes

Not ready for detailed review, yet. But two high-level questions around ABI stability could already be discussed:

  1. How to make sure we still emit the ~exception_ptr and copy constructor/assignment in the library. The current approach was previously discussed in https://reviews.llvm.org/D122536 (among others by @philnik777). But if there is a better approach, I am happy to adjust this PR
  2. If we actually want to introduce the new __do_{inc,dec}rement functions or if we want to directly call __cxa_{inc,dec}rement_refcount from the headers

This commit adds an inlined fast-path for nullptrs to the copy constructor, copy assignment and destructor of std::exception_ptr.

Performance results (from libc++'s CI):

Benchmark                               Baseline    Candidate    Difference    % Difference
------------------------------------  ----------  -----------  ------------  --------------
bm_exception_ptr_copy_assign_nonnull       10.00         7.49         -2.51         -25.11%
bm_exception_ptr_copy_assign_null          10.30         0.94         -9.36         -90.91%
bm_exception_ptr_copy_ctor_nonnull          7.52         6.91         -0.61          -8.16%
bm_exception_ptr_copy_ctor_null            11.24         0.62        -10.61         -94.46%
bm_exception_ptr_move_assign_nonnull       14.32         7.20         -7.12         -49.73%
bm_exception_ptr_move_assign_null          10.90         0.94         -9.96         -91.39%
bm_exception_ptr_move_ctor_nonnull         10.95         6.87         -4.08         -37.24%
bm_exception_ptr_move_ctor_null             7.55         0.94         -6.61         -87.57%
bm_exception_ptr_swap_nonnull               0.62         0.62          0.00           0.08%
bm_exception_ptr_swap_null                  7.82         0.94         -6.88         -88.01%

Full diff: https://github.com/llvm/llvm-project/pull/165909.diff

5 Files Affected:

  • (modified) libcxx/include/__exception/exception_ptr.h (+60-5)
  • (modified) libcxx/src/exception.cpp (+1)
  • (modified) libcxx/src/support/runtime/exception_pointer_cxxabi.ipp (+4-11)
  • (modified) libcxx/src/support/runtime/exception_pointer_glibcxx.ipp (+10-17)
  • (modified) libcxx/src/support/runtime/exception_pointer_unimplemented.ipp (+2-7)
diff --git a/libcxx/include/__exception/exception_ptr.h b/libcxx/include/__exception/exception_ptr.h
index e78126ea23852..0f081c6bc8ed0 100644
--- a/libcxx/include/__exception/exception_ptr.h
+++ b/libcxx/include/__exception/exception_ptr.h
@@ -30,6 +30,28 @@ _LIBCPP_PUSH_MACROS
 
 #ifndef _LIBCPP_ABI_MICROSOFT
 
+// Previously, parts of exception_ptr were defined out-of-line, which prevented
+// useful compiler optimizations. Changing the out-of-line definitions to inline
+// definitions is an ABI break, however. To prevent this, we have to make sure
+// the symbols remain available in the libc++ library, in addition to being
+// defined inline here in this header.
+// To this end, we use _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE macro:
+// The macro is defined as empty for src/exception.cpp, forcing the definitions of
+// the functions to be emitted and included in the library. When users of libc++
+// compile their code, the __gnu_inline__ attribute will suppress generation of
+// these functions while making their definitions available for inlining.
+#  ifdef _LIBCPP_EMIT_CODE_FOR_EXCEPTION_PTR
+#    define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE _LIBCPP_EXPORTED_FROM_ABI
+#  else
+#    if !__has_cpp_attribute(__gnu__::__gnu_inline__)
+#      error "GNU inline attribute is not supported"
+#    endif
+#    define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE [[__gnu__::__gnu_inline__]] inline
+#  endif
+
+_LIBCPP_DIAGNOSTIC_PUSH
+_LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wgnu-inline-cpp-without-extern")
+
 #  if _LIBCPP_AVAILABILITY_HAS_INIT_PRIMARY_EXCEPTION
 
 namespace __cxxabiv1 {
@@ -67,6 +89,17 @@ inline _LIBCPP_HIDE_FROM_ABI void swap(exception_ptr& __x, exception_ptr& __y) _
 class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   void* __ptr_;
 
+  static void __do_increment_refcount(void* __ptr) _NOEXCEPT;
+  static void __do_decrement_refcount(void* __ptr) _NOEXCEPT;
+  _LIBCPP_HIDE_FROM_ABI static void __increment_refcount(void* __ptr) _NOEXCEPT {
+    if (__ptr)
+      __do_increment_refcount(__ptr);
+  }
+  _LIBCPP_HIDE_FROM_ABI static void __decrement_refcount(void* __ptr) _NOEXCEPT {
+    if (__ptr)
+      __do_decrement_refcount(__ptr);
+  }
+
   static exception_ptr __from_native_exception_pointer(void*) _NOEXCEPT;
 
   template <class _Ep>
@@ -81,17 +114,18 @@ class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   _LIBCPP_HIDE_FROM_ABI exception_ptr() _NOEXCEPT : __ptr_() {}
   _LIBCPP_HIDE_FROM_ABI exception_ptr(nullptr_t) _NOEXCEPT : __ptr_() {}
 
-  exception_ptr(const exception_ptr&) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr(const exception_ptr&) _NOEXCEPT;
   _LIBCPP_HIDE_FROM_ABI exception_ptr(exception_ptr&& __other) _NOEXCEPT : __ptr_(__other.__ptr_) {
     __other.__ptr_ = nullptr;
   }
-  exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
   _LIBCPP_HIDE_FROM_ABI exception_ptr& operator=(exception_ptr&& __other) _NOEXCEPT {
-    exception_ptr __tmp(std::move(__other));
-    std::swap(__tmp, *this);
+    __decrement_refcount(__ptr_);
+    __ptr_         = __other.__ptr_;
+    __other.__ptr_ = nullptr;
     return *this;
   }
-  ~exception_ptr() _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE ~exception_ptr() _NOEXCEPT;
 
   _LIBCPP_HIDE_FROM_ABI explicit operator bool() const _NOEXCEPT { return __ptr_ != nullptr; }
 
@@ -109,6 +143,25 @@ class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   friend _LIBCPP_EXPORTED_FROM_ABI void rethrow_exception(exception_ptr);
 };
 
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr::exception_ptr(const exception_ptr& __other) _NOEXCEPT
+    : __ptr_(__other.__ptr_) {
+  __increment_refcount(__ptr_);
+}
+
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& exception_ptr::operator=(const exception_ptr& __other) _NOEXCEPT {
+  if (__ptr_ != __other.__ptr_) {
+    __increment_refcount(__other.__ptr_);
+    __decrement_refcount(__ptr_);
+    __ptr_ = __other.__ptr_;
+  }
+  return *this;
+}
+
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr::~exception_ptr() _NOEXCEPT { __decrement_refcount(__ptr_); }
+
 inline _LIBCPP_HIDE_FROM_ABI void swap(exception_ptr& __x, exception_ptr& __y) _NOEXCEPT {
   std::swap(__x.__ptr_, __y.__ptr_);
 }
@@ -222,6 +275,8 @@ _LIBCPP_HIDE_FROM_ABI exception_ptr make_exception_ptr(_Ep __e) _NOEXCEPT {
 #endif // _LIBCPP_ABI_MICROSOFT
 _LIBCPP_END_UNVERSIONED_NAMESPACE_STD
 
+_LIBCPP_DIAGNOSTIC_POP
+
 _LIBCPP_POP_MACROS
 
 #endif // _LIBCPP___EXCEPTION_EXCEPTION_PTR_H
diff --git a/libcxx/src/exception.cpp b/libcxx/src/exception.cpp
index ac6324cd9fe35..9dd9b0c9938fd 100644
--- a/libcxx/src/exception.cpp
+++ b/libcxx/src/exception.cpp
@@ -8,6 +8,7 @@
 
 #define _LIBCPP_ENABLE_CXX20_REMOVED_UNCAUGHT_EXCEPTION
 #define _LIBCPP_DISABLE_DEPRECATION_WARNINGS
+#define _LIBCPP_EMIT_CODE_FOR_EXCEPTION_PTR
 
 #include <exception>
 #include <new>
diff --git a/libcxx/src/support/runtime/exception_pointer_cxxabi.ipp b/libcxx/src/support/runtime/exception_pointer_cxxabi.ipp
index 8f5c2060bb06c..e09bf8981263f 100644
--- a/libcxx/src/support/runtime/exception_pointer_cxxabi.ipp
+++ b/libcxx/src/support/runtime/exception_pointer_cxxabi.ipp
@@ -13,19 +13,12 @@
 
 namespace std {
 
-exception_ptr::~exception_ptr() noexcept { __cxa_decrement_exception_refcount(__ptr_); }
-
-exception_ptr::exception_ptr(const exception_ptr& other) noexcept : __ptr_(other.__ptr_) {
-  __cxa_increment_exception_refcount(__ptr_);
+void exception_ptr::__do_increment_refcount(void* __ptr) noexcept {
+  __cxa_increment_exception_refcount(__ptr);
 }
 
-exception_ptr& exception_ptr::operator=(const exception_ptr& other) noexcept {
-  if (__ptr_ != other.__ptr_) {
-    __cxa_increment_exception_refcount(other.__ptr_);
-    __cxa_decrement_exception_refcount(__ptr_);
-    __ptr_ = other.__ptr_;
-  }
-  return *this;
+void exception_ptr::__do_decrement_refcount(void* __ptr) noexcept {
+  __cxa_decrement_exception_refcount(__ptr);
 }
 
 exception_ptr exception_ptr::__from_native_exception_pointer(void* __e) noexcept {
diff --git a/libcxx/src/support/runtime/exception_pointer_glibcxx.ipp b/libcxx/src/support/runtime/exception_pointer_glibcxx.ipp
index 174b44ce0e6f7..c7b2e343b5f09 100644
--- a/libcxx/src/support/runtime/exception_pointer_glibcxx.ipp
+++ b/libcxx/src/support/runtime/exception_pointer_glibcxx.ipp
@@ -7,14 +7,14 @@
 //
 //===----------------------------------------------------------------------===//
 
+
 // libsupc++ does not implement the dependent EH ABI and the functionality
 // it uses to implement std::exception_ptr (which it declares as an alias of
 // std::__exception_ptr::exception_ptr) is not directly exported to clients. So
 // we have little choice but to hijack std::__exception_ptr::exception_ptr's
-// (which fortunately has the same layout as our std::exception_ptr) copy
-// constructor, assignment operator and destructor (which are part of its
-// stable ABI), and its rethrow_exception(std::__exception_ptr::exception_ptr)
-// function.
+// _M_addref and _M_release and its rethrow_exception function. Fortunately,
+// glibcxx's exception_ptr has the same layout as our exception_ptr and we can
+// reinterpret_cast between the two.
 
 namespace std {
 
@@ -23,27 +23,20 @@ namespace __exception_ptr {
 struct exception_ptr {
   void* __ptr_;
 
-  explicit exception_ptr(void*) noexcept;
-  exception_ptr(const exception_ptr&) noexcept;
-  exception_ptr& operator=(const exception_ptr&) noexcept;
-  ~exception_ptr() noexcept;
+  void _M_addref() noexcept;
+  void _M_release() noexcept;
 };
 
 } // namespace __exception_ptr
 
 [[noreturn]] void rethrow_exception(__exception_ptr::exception_ptr);
 
-exception_ptr::~exception_ptr() noexcept { reinterpret_cast<__exception_ptr::exception_ptr*>(this)->~exception_ptr(); }
-
-exception_ptr::exception_ptr(const exception_ptr& other) noexcept : __ptr_(other.__ptr_) {
-  new (reinterpret_cast<void*>(this))
-      __exception_ptr::exception_ptr(reinterpret_cast<const __exception_ptr::exception_ptr&>(other));
+void exception_ptr::__do_increment_refcount(void* __ptr) noexcept {
+  reinterpret_cast<__exception_ptr::exception_ptr*>(this)->_M_addref();
 }
 
-exception_ptr& exception_ptr::operator=(const exception_ptr& other) noexcept {
-  *reinterpret_cast<__exception_ptr::exception_ptr*>(this) =
-      reinterpret_cast<const __exception_ptr::exception_ptr&>(other);
-  return *this;
+void exception_ptr::__do_decrement_refcount(void* __ptr) noexcept {
+  reinterpret_cast<__exception_ptr::exception_ptr*>(this)->_M_release();
 }
 
 exception_ptr exception_ptr::__from_native_exception_pointer(void* __e) noexcept {
diff --git a/libcxx/src/support/runtime/exception_pointer_unimplemented.ipp b/libcxx/src/support/runtime/exception_pointer_unimplemented.ipp
index 05a71ce34e5ac..78be16bf95188 100644
--- a/libcxx/src/support/runtime/exception_pointer_unimplemented.ipp
+++ b/libcxx/src/support/runtime/exception_pointer_unimplemented.ipp
@@ -11,17 +11,12 @@
 
 namespace std {
 
-exception_ptr::~exception_ptr() noexcept {
+void exception_ptr::__do_increment_refcount(void* __ptr) noexcept {
 #warning exception_ptr not yet implemented
   __libcpp_verbose_abort("exception_ptr not yet implemented\n");
 }
 
-exception_ptr::exception_ptr(const exception_ptr& other) noexcept : __ptr_(other.__ptr_) {
-#warning exception_ptr not yet implemented
-  __libcpp_verbose_abort("exception_ptr not yet implemented\n");
-}
-
-exception_ptr& exception_ptr::operator=(const exception_ptr& other) noexcept {
+void exception_ptr::__do_decrement_refcount(void* __ptr) noexcept {
 #warning exception_ptr not yet implemented
   __libcpp_verbose_abort("exception_ptr not yet implemented\n");
 }

@vogelsgesang
Copy link
Member Author

vogelsgesang commented Nov 4, 2025

Do you think this is ready for review?

The PR is ready for a first round of feedback, in particular on the high-level questions:

  1. How to make sure we still emit the ~exception_ptr and copy constructor/assignment in the library? (The current approach was previously discussed in https://reviews.llvm.org/D122536)
  2. Do we actually want to introduce the new __do_{inc,dec}rement functions or do want to directly call __cxa_{inc,dec}rement_refcount from the headers?

The CI is still red since I did not yet update the ABI list and because I did not yet add the new _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE macro to the clang-tidy check. I still need to address those, but would already appreciate feedback on those two higher-level points By now the CI should also be green

@vogelsgesang vogelsgesang force-pushed the avogelsgesang-exception-ptr-inline branch from e06f411 to 7684774 Compare November 4, 2025 11:41
Copy link
Contributor

@philnik777 philnik777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, please don't tag anybody in the commit message. That will cause notifications any time the commit is pushed anywhere on github otherwise.

Comment on lines +33 to +50
// Previously, parts of exception_ptr were defined out-of-line, which prevented
// useful compiler optimizations. Changing the out-of-line definitions to inline
// definitions is an ABI break, however. To prevent this, we have to make sure
// the symbols remain available in the libc++ library, in addition to being
// defined inline here in this header.
// To this end, we use _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE macro:
// The macro is defined as empty for src/exception.cpp, forcing the definitions of
// the functions to be emitted and included in the library. When users of libc++
// compile their code, the __gnu_inline__ attribute will suppress generation of
// these functions while making their definitions available for inlining.
# ifdef _LIBCPP_EMIT_CODE_FOR_EXCEPTION_PTR
# define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE _LIBCPP_EXPORTED_FROM_ABI
# else
# if !__has_cpp_attribute(__gnu__::__gnu_inline__)
# error "GNU inline attribute is not supported"
# endif
# define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE [[__gnu__::__gnu_inline__]] inline
# endif
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop this entire thing and instead just have an #ifdef _LIBCPP_BUILDING_LIBRARY, in which case we provide the old declarations, and otherwise provide inline functions. The inline functions get different mangling anyways, so I don't think there is much of a problem. I have actually no idea why I suggested this convoluted workaround before.

Comment on lines +94 to +101
_LIBCPP_HIDE_FROM_ABI static void __increment_refcount(void* __ptr) _NOEXCEPT {
if (__ptr)
__do_increment_refcount(__ptr);
}
_LIBCPP_HIDE_FROM_ABI static void __decrement_refcount(void* __ptr) _NOEXCEPT {
if (__ptr)
__do_decrement_refcount(__ptr);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like these names are rather poor. It'd either inline these relatively simple functions or rename them so it's obvious they might inc/decrement the refcount.

Comment on lines +92 to +93
static void __do_increment_refcount(void* __ptr) _NOEXCEPT;
static void __do_decrement_refcount(void* __ptr) _NOEXCEPT;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static void __do_increment_refcount(void* __ptr) _NOEXCEPT;
static void __do_decrement_refcount(void* __ptr) _NOEXCEPT;
static void __do_increment_refcount([[__gnu__::__nonnull__]] _LIBCPP_NOESCAPE void*) _NOEXCEPT;
static void __do_decrement_refcount([[__gnu__::__nonnull__]] _LIBCPP_NOESCAPE void*) _NOEXCEPT;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants