[libc++] Optimize std::exception_ptr

This commit optimizes the performance for `std::exception_ptr`
for an empty exception objects.

To do so, we use 3 high-level approaches:
1. Moving the implementation from the libc++ library into the libc++
   headers, thereby allowing the compiler to inline the function bodies.
2. Adding fast paths to the (now inlineable) bodies, checking for the
   empty case.
3. Adding move constuctor, assignment and `swap`

Those optimizations were implemented for the libc++abi, libsupc++ and
libstdc++ ABIs.

Fixes #XXX

Performance
-----------

With this change, the compiler can now completely constant-fold
https://godbolt.org/z/NaNKe5. Also in cases where the compiler cannot
statically prove that exception_ptr is empty, it can at least generate a
fast-path check if the exception_ptr is empty, without calling into the
library.

ABI compatibility
-----------------

We use a new visibility macro `_LIBCPP_EXPORTED_FROM_ABI_INLINEABLE` to
ensure that the functions which are now declared in the header are still
exported by the library. See the description in `VisibilityMacros.rst`
for details. This approach was originally proposed by Nikolas Klauser
in https://reviews.llvm.org/D122536

Moving implementations to the header
------------------------------------

To move the implementation to the header, we must know the selected
LIBCXX_CXX_ABI also in the headers. For that purpose, the LIBCXX_CXX_ABI
configuration is now exposed via `__config_site`.

While the Microsoft ABI and the "unimplemented" APIs do not benefit
from this optimizations, I also moved their implementation to the
headers for more uniformity. Mid-term, we probably have to expose all
implementations in the header, anyway, because P3068R6 mandates all
methods of `exception_ptr` to be constexpr.

Unifying libc++abi, libsupc++ and `none`
-----------------------------------------------

Both libc++abi and libsupc++ are reference-counted. The primary
difference is the function called for ref-counting:
* libc++api uses `__cxa_{in,de}crement_exception_refcount`
* libsupc++ uses `__exception_ptr::_M_{addref,release}`

This commit factors out the common reference-counting logic into a
shared header.

Our libsupc++ implementation so far did not take advantage of
`_M_addref`/`_M_release`. For the performance benefits of this PR it was
necessary to start using them.

The same reference-counted approach is also reused for the
`none`/`unimplemented` implementation. This has the side effect, that
users can now use empty `exception_ptr`s even in the `none` ABI. The
abort is only triggered for non-empty `exception_ptr`s. Given this
simplifies the code, I think change is acceptable.

Unifying nested_exception
-------------------------

The implementation for `nested_exception` was effectively identical
across all ABIs. For most ABIs, the source code was literally identical.
There were only two deviations:
* For libsupc++ or libstdc++, we did not use define the
  `~nested_exception` destructor.
* The abort in `nested_exception::rethrow_nested` was unnecessary as the
  same abort is also present in the `rethrow_exception` implementation.
  As such, we were able to simply remove that special casing.

The implementation is now unified directly in the `__nested_exception.h`
header.

Standard conformance
--------------------

The available constructors, operators and methods of `exception_ptr` is
not specified by the standard. As such, adding the move constructor and
the assignment operator are standard conformant. libstdc++ made the same
decision and also provides those headers.

Author: vogelsgesang

libcxx/CMakeLists.txt

@@ -731,6 +731,7 @@ config_define(${LIBCXX_ABI_VERSION} _LIBCPP_ABI_VERSION)
 config_define(${LIBCXX_ABI_NAMESPACE} _LIBCPP_ABI_NAMESPACE)
 config_define(${LIBCXX_ABI_FORCE_ITANIUM} _LIBCPP_ABI_FORCE_ITANIUM)
 config_define(${LIBCXX_ABI_FORCE_MICROSOFT} _LIBCPP_ABI_FORCE_MICROSOFT)
+config_define(${LIBCXX_CXX_ABI} _LIBCXX_CXX_ABI)
 config_define(${LIBCXX_ENABLE_THREADS} _LIBCPP_HAS_THREADS)
 config_define(${LIBCXX_ENABLE_MONOTONIC_CLOCK} _LIBCPP_HAS_MONOTONIC_CLOCK)
 config_define(${LIBCXX_HAS_TERMINAL_AVAILABLE} _LIBCPP_HAS_TERMINAL)
@@ -750,6 +751,20 @@ config_define(${LIBCXX_ENABLE_WIDE_CHARACTERS} _LIBCPP_HAS_WIDE_CHARACTERS)
 config_define(${LIBCXX_ENABLE_TIME_ZONE_DATABASE} _LIBCPP_HAS_TIME_ZONE_DATABASE)
 config_define(${LIBCXX_ENABLE_VENDOR_AVAILABILITY_ANNOTATIONS} _LIBCPP_HAS_VENDOR_AVAILABILITY_ANNOTATIONS)
 
+if (LIBCXX_CXX_ABI STREQUAL "none")
+  config_define(1 _LIBCPP_CXX_ABI_NONE)
+elseif (LIBCXX_CXX_ABI STREQUAL "libcxxabi" OR LIBCXX_CXX_ABI STREQUAL "system-libcxxabi")
+  config_define(1 _LIBCPP_CXX_ABI_LIBCXXABI)
+elseif (LIBCXX_CXX_ABI STREQUAL "libcxxrt")
+  config_define(1 _LIBCPP_CXX_ABI_LIBCXXRT)
+elseif (LIBCXX_CXX_ABI STREQUAL "libstdc++")
+  config_define(1 _LIBCPP_CXX_ABI_LIBSTDCXX)
+elseif (LIBCXX_CXX_ABI STREQUAL "libsupc++")
+  config_define(1 _LIBCPP_CXX_ABI_LIBSUPCXX)
+elseif (LIBCXX_CXX_ABI STREQUAL "vcruntime")
+  config_define(1 _LIBCPP_CXX_ABI_VCRUNTIME)
+endif()
+
 # TODO: Remove in LLVM 21. We're leaving an error to make this fail explicitly.
 if (LIBCXX_ENABLE_ASSERTIONS)
   message(FATAL_ERROR "LIBCXX_ENABLE_ASSERTIONS has been removed. Please use LIBCXX_HARDENING_MODE instead.")

libcxx/cmake/Modules/HandleLibCXXABI.cmake

@@ -119,7 +119,6 @@ elseif ("${LIBCXX_CXX_ABI}" STREQUAL "libsupc++")
 elseif ("${LIBCXX_CXX_ABI}" STREQUAL "libcxxabi")
   add_library(libcxx-abi-headers INTERFACE)
   target_link_libraries(libcxx-abi-headers INTERFACE cxxabi-headers)
-  target_compile_definitions(libcxx-abi-headers INTERFACE "-DLIBCXX_BUILDING_LIBCXXABI")
 
   if (TARGET cxxabi_shared)
     add_library(libcxx-abi-shared INTERFACE)
@@ -156,7 +155,6 @@ elseif ("${LIBCXX_CXX_ABI}" STREQUAL "system-libcxxabi")
 
   add_library(libcxx-abi-headers INTERFACE)
   import_private_headers(libcxx-abi-headers "${LIBCXX_CXX_ABI_INCLUDE_PATHS}" "cxxabi.h;__cxxabi_config.h")
-  target_compile_definitions(libcxx-abi-headers INTERFACE "-DLIBCXX_BUILDING_LIBCXXABI")
 
   import_shared_library(libcxx-abi-shared c++abi)
   target_link_libraries(libcxx-abi-shared INTERFACE libcxx-abi-headers)
@@ -173,7 +171,6 @@ elseif ("${LIBCXX_CXX_ABI}" STREQUAL "libcxxrt")
   add_library(libcxx-abi-headers INTERFACE)
   import_private_headers(libcxx-abi-headers "${LIBCXX_CXX_ABI_INCLUDE_PATHS}"
     "cxxabi.h;unwind.h;unwind-arm.h;unwind-itanium.h")
-  target_compile_definitions(libcxx-abi-headers INTERFACE "-DLIBCXXRT")
 
   import_shared_library(libcxx-abi-shared cxxrt)
   target_link_libraries(libcxx-abi-shared INTERFACE libcxx-abi-headers)
@@ -191,7 +188,6 @@ elseif ("${LIBCXX_CXX_ABI}" STREQUAL "vcruntime")
 # Don't link against any ABI library
 elseif ("${LIBCXX_CXX_ABI}" STREQUAL "none")
   add_library(libcxx-abi-headers INTERFACE)
-  target_compile_definitions(libcxx-abi-headers INTERFACE "-D_LIBCPP_BUILDING_HAS_NO_ABI_LIBRARY")
 
   add_library(libcxx-abi-shared INTERFACE)
   target_link_libraries(libcxx-abi-shared INTERFACE libcxx-abi-headers)

libcxx/docs/DesignDocs/VisibilityMacros.rst

@@ -35,6 +35,25 @@ Visibility Macros
   used on class templates. On classes it should only be used if the vtable
   lives in the built library.
 
+**_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE**
+  Mark a symbol as exported from the libc++ library, while still providing an
+  inlineable definition that can be used by the compiler for optimization
+  purposes.
+
+  To use this macro on a class method, define the method body
+  *outside* of the class definition and annotate that definition with
+  `_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE`. Make sure to include the
+  header in at least one translation unit linked into the libc++ library.
+
+  This macro works by applying `[[gnu::gnu_inline]] inline` to the funciton
+  in the header, thereby suppressing code generation while still allowing the
+  compiler to use the function for optimization purposes.
+  During the build of libc++, we trigger code generation by not expanding the
+  macro to `_LIBCPP_EXPORTED_FROM_ABI`. Since the function is no longer marked
+  as `inline`, it will be emitted even if not called. (For this reason its
+  paramount to not define methods in the class definition, since those definitions
+  would be implicitly `inline`.)
+
 **_LIBCPP_OVERRIDABLE_FUNC_VIS**
   Mark a symbol as being exported by the libc++ library, but allow it to be
   overridden locally. On non-Windows, this is equivalent to `_LIBCPP_FUNC_VIS`.

libcxx/include/CMakeLists.txt

@@ -343,6 +343,10 @@ set(files
   __debug_utils/strict_weak_ordering_check.h
   __exception/exception.h
   __exception/exception_ptr.h
+  __exception/exception_ptr_cxxabi.ipp
+  __exception/exception_ptr_glibcxx.ipp
+  __exception/exception_ptr_msvc.ipp
+  __exception/exception_ptr_unimplemented.ipp
   __exception/nested_exception.h
   __exception/operations.h
   __exception/terminate.h

libcxx/include/__config_site.in

@@ -33,6 +33,14 @@
 #cmakedefine01 _LIBCPP_HAS_TIME_ZONE_DATABASE
 #cmakedefine01 _LIBCPP_INSTRUMENTED_WITH_ASAN
 
+// LIBCXX_CXX_ABI backends
+#cmakedefine _LIBCPP_CXX_ABI_NONE
+#cmakedefine _LIBCPP_CXX_ABI_LIBCXXABI
+#cmakedefine _LIBCPP_CXX_ABI_LIBCXXRT
+#cmakedefine _LIBCPP_CXX_ABI_LIBSTDCXX
+#cmakedefine _LIBCPP_CXX_ABI_LIBSUPCXX
+#cmakedefine _LIBCPP_CXX_ABI_VCRUNTIME
+
 // PSTL backends
 #cmakedefine _LIBCPP_PSTL_BACKEND_SERIAL
 #cmakedefine _LIBCPP_PSTL_BACKEND_STD_THREAD

libcxx/include/__exception/exception_ptr.h

@@ -23,9 +23,29 @@
 #  pragma GCC system_header
 #endif
 
-#ifndef _LIBCPP_ABI_MICROSOFT
+// Previously, parts of exception_ptr were defined out-of-line, which prevented
+// useful compiler optimizations. Changing the out-of-line definitions to inline
+// definitions is an ABI break, however. To prevent this, we have to make sure
+// the symbols remain available in the libc++ library, in addition to being
+// defined inline here in this header.
+// To this end, we use _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE macro:
+// The macro is defined as empty for src/exception.cpp, forcing the definitions of
+// the functions to be emitted and included in the library. When users of libc++
+// compile their code, the __gnu_inline__ attribute will suppress generation of
+// these functions while making their definitions available for inlining.
+#  ifdef _LIBCPP_EMIT_CODE_FOR_EXCEPTION_PTR
+#    define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE _LIBCPP_EXPORTED_FROM_ABI
+#  else
+#    if !__has_cpp_attribute(gnu::gnu_inline)
+#      error "GNU inline attribute is not supported"
+#    endif
+#    define _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE [[gnu::gnu_inline]] inline
+#  endif
 
-#  if _LIBCPP_AVAILABILITY_HAS_INIT_PRIMARY_EXCEPTION
+_LIBCPP_DIAGNOSTIC_PUSH
+_LIBCPP_CLANG_DIAGNOSTIC_IGNORED("-Wgnu-inline-cpp-without-extern")
+
+#ifdef _LIBCPP_AVAILABILITY_HAS_INIT_PRIMARY_EXCEPTION
 
 namespace __cxxabiv1 {
 
@@ -49,18 +69,25 @@ _LIBCPP_OVERRIDABLE_FUNC_VIS __cxa_exception* __cxa_init_primary_exception(
 
 } // namespace __cxxabiv1
 
-#  endif
-
 #endif
 
 _LIBCPP_BEGIN_UNVERSIONED_NAMESPACE_STD
 
-#ifndef _LIBCPP_ABI_MICROSOFT
+class _LIBCPP_EXPORTED_FROM_ABI exception_ptr;
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr current_exception() _NOEXCEPT;
+[[__noreturn__]] _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE void rethrow_exception(exception_ptr);
+
+#ifndef _LIBCPP_CXX_ABI_VCRUNTIME
 
 class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   void* __ptr_;
 
-  static exception_ptr __from_native_exception_pointer(void*) _NOEXCEPT;
+  // Customization points to adjust the reference counting for cxxabi or
+  // libsupc++/libstdc++
+  _LIBCPP_HIDE_FROM_ABI _LIBCPP_ALWAYS_INLINE static void __increment_refcount(void* __ptr) _NOEXCEPT;
+  _LIBCPP_HIDE_FROM_ABI _LIBCPP_ALWAYS_INLINE static void __decrement_refcount(void* __ptr) _NOEXCEPT;
+
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE static exception_ptr __from_native_exception_pointer(void*) _NOEXCEPT;
 
   template <class _Ep>
   friend _LIBCPP_HIDE_FROM_ABI exception_ptr __make_exception_ptr_explicit(_Ep&) _NOEXCEPT;
@@ -74,9 +101,11 @@ class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   _LIBCPP_HIDE_FROM_ABI exception_ptr() _NOEXCEPT : __ptr_() {}
   _LIBCPP_HIDE_FROM_ABI exception_ptr(nullptr_t) _NOEXCEPT : __ptr_() {}
 
-  exception_ptr(const exception_ptr&) _NOEXCEPT;
-  exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
-  ~exception_ptr() _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr(const exception_ptr&) _NOEXCEPT;
+  _LIBCPP_HIDE_FROM_ABI exception_ptr(exception_ptr&&) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& operator=(const exception_ptr&) _NOEXCEPT;
+  _LIBCPP_HIDE_FROM_ABI exception_ptr& operator=(exception_ptr&&) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE ~exception_ptr() _NOEXCEPT;
 
   _LIBCPP_HIDE_FROM_ABI explicit operator bool() const _NOEXCEPT { return __ptr_ != nullptr; }
 
@@ -88,10 +117,53 @@ class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
     return !(__x == __y);
   }
 
+  friend _LIBCPP_HIDE_FROM_ABI void swap(exception_ptr& __x, exception_ptr& __y) {
+    void* __tmp = __x.__ptr_;
+    __x.__ptr_ = __y.__ptr_;
+    __y.__ptr_ = __tmp;
+  }
+
   friend _LIBCPP_EXPORTED_FROM_ABI exception_ptr current_exception() _NOEXCEPT;
   friend _LIBCPP_EXPORTED_FROM_ABI void rethrow_exception(exception_ptr);
 };
 
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr exception_ptr::__from_native_exception_pointer(void* __e) noexcept {
+  __increment_refcount(__e);
+  exception_ptr __ptr;
+  __ptr.__ptr_ = __e;
+  return __ptr;
+}
+
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr::exception_ptr(const exception_ptr& __other) noexcept : __ptr_(__other.__ptr_) {
+  __increment_refcount(__ptr_);
+}
+
+_LIBCPP_HIDE_FROM_ABI inline exception_ptr::exception_ptr(exception_ptr&& __other) _NOEXCEPT : __ptr_(__other.__ptr_) {
+  __other.__ptr_ = nullptr;
+}
+
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& exception_ptr::operator=(const exception_ptr& __other) noexcept {
+  if (__ptr_ != __other.__ptr_) {
+    __increment_refcount(__other.__ptr_);
+    __decrement_refcount(__ptr_);
+    __ptr_ = __other.__ptr_;
+  }
+  return *this;
+}
+
+_LIBCPP_HIDE_FROM_ABI inline exception_ptr& exception_ptr::operator=(exception_ptr&& __other) noexcept {
+  __decrement_refcount(__ptr_);
+  __ptr_ = __other.__ptr_;
+  __other.__ptr_ = nullptr;
+  return *this;
+}
+
+// Must be defined outside the class definition due to _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr::~exception_ptr() noexcept { __decrement_refcount(__ptr_); }
+
 #  if _LIBCPP_HAS_EXCEPTIONS
 #    if _LIBCPP_AVAILABILITY_HAS_INIT_PRIMARY_EXCEPTION
 template <class _Ep>
@@ -167,26 +239,26 @@ class _LIBCPP_EXPORTED_FROM_ABI exception_ptr {
   _LIBCPP_DIAGNOSTIC_POP
 
 public:
-  exception_ptr() _NOEXCEPT;
-  exception_ptr(nullptr_t) _NOEXCEPT;
-  exception_ptr(const exception_ptr& __other) _NOEXCEPT;
-  exception_ptr& operator=(const exception_ptr& __other) _NOEXCEPT;
-  exception_ptr& operator=(nullptr_t) _NOEXCEPT;
-  ~exception_ptr() _NOEXCEPT;
-  explicit operator bool() const _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr() _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr(nullptr_t) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr(const exception_ptr& __other) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& operator=(const exception_ptr& __other) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr& operator=(nullptr_t) _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE ~exception_ptr() _NOEXCEPT;
+  _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE explicit operator bool() const _NOEXCEPT;
 };
 
-_LIBCPP_EXPORTED_FROM_ABI bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT;
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE bool operator==(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT;
 
 inline _LIBCPP_HIDE_FROM_ABI bool operator!=(const exception_ptr& __x, const exception_ptr& __y) _NOEXCEPT {
   return !(__x == __y);
 }
 
-_LIBCPP_EXPORTED_FROM_ABI void swap(exception_ptr&, exception_ptr&) _NOEXCEPT;
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE void swap(exception_ptr&, exception_ptr&) _NOEXCEPT;
 
-_LIBCPP_EXPORTED_FROM_ABI exception_ptr __copy_exception_ptr(void* __except, const void* __ptr);
-_LIBCPP_EXPORTED_FROM_ABI exception_ptr current_exception() _NOEXCEPT;
-[[__noreturn__]] _LIBCPP_EXPORTED_FROM_ABI void rethrow_exception(exception_ptr);
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr __copy_exception_ptr(void* __except, const void* __ptr);
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr current_exception() _NOEXCEPT;
+[[__noreturn__]] _LIBCPP_EXPORTED_FROM_LIB_INLINEABLE void rethrow_exception(exception_ptr);
 
 // This is a built-in template function which automagically extracts the required
 // information.
@@ -199,6 +271,21 @@ _LIBCPP_HIDE_FROM_ABI exception_ptr make_exception_ptr(_Ep __e) _NOEXCEPT {
 }
 
 #endif // _LIBCPP_ABI_MICROSOFT
+
 _LIBCPP_END_UNVERSIONED_NAMESPACE_STD
 
+#  if defined(_LIBCPP_CXX_ABI_NONE)
+#    include <__exception/exception_ptr_unimplemented.ipp>
+#  elif defined(_LIBCPP_CXX_ABI_LIBCXXABI) || defined(_LIBCPP_CXX_ABI_LIBCXXRT)
+#    include <__exception/exception_ptr_cxxabi.ipp>
+#  elif defined(_LIBCPP_CXX_ABI_LIBSTDCXX) || defined(_LIBCPP_CXX_ABI_LIBSUPCXX)
+#    include <__exception/exception_ptr_glibcxx.ipp>
+#  elif defined(_LIBCPP_CXX_ABI_VCRUNTIME)
+#    include <__exception/exception_ptr_msvc.ipp>
+#  else
+#    error "Unsupported C++ ABI library"
+#  endif
+
+_LIBCPP_DIAGNOSTIC_POP
+
 #endif // _LIBCPP___EXCEPTION_EXCEPTION_PTR_H

libcxx/include/__exception/exception_ptr_cxxabi.ipp

@@ -0,0 +1,50 @@
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <__exception/terminate.h>
+
+namespace __cxxabiv1 {
+
+extern "C" {
+_LIBCPP_OVERRIDABLE_FUNC_VIS void __cxa_increment_exception_refcount(void*) noexcept;
+_LIBCPP_OVERRIDABLE_FUNC_VIS void __cxa_decrement_exception_refcount(void*) noexcept;
+_LIBCPP_OVERRIDABLE_FUNC_VIS void* __cxa_current_primary_exception() noexcept;
+_LIBCPP_OVERRIDABLE_FUNC_VIS void __cxa_rethrow_primary_exception(void*);
+}
+  
+} // namespace __cxxabiv1
+
+namespace std {
+
+_LIBCPP_HIDE_FROM_ABI _LIBCPP_ALWAYS_INLINE inline void exception_ptr::__increment_refcount(void* __ptr) _NOEXCEPT {
+  if (__ptr)
+    __cxxabiv1::__cxa_increment_exception_refcount(__ptr);
+}
+
+_LIBCPP_HIDE_FROM_ABI _LIBCPP_ALWAYS_INLINE inline void exception_ptr::__decrement_refcount(void* __ptr) _NOEXCEPT {
+  if (__ptr)
+    __cxxabiv1::__cxa_decrement_exception_refcount(__ptr);
+}
+
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE exception_ptr current_exception() _NOEXCEPT {
+  // It would be nicer if there was a constructor that took a ptr, then
+  // this whole function would be just:
+  //    return exception_ptr(__cxa_current_primary_exception());
+  exception_ptr __ptr;
+  __ptr.__ptr_ =  __cxxabiv1::__cxa_current_primary_exception();
+  return __ptr;
+}
+
+_LIBCPP_EXPORTED_FROM_LIB_INLINEABLE void rethrow_exception(exception_ptr __ptr) {
+  __cxxabiv1::__cxa_rethrow_primary_exception(__ptr.__ptr_);
+  // if __ptr.__ptr_ is NULL, above returns so we terminate.
+  terminate();
+}
+
+} // namespace std