[Bolt][aarch64] Some problems encountered when running shared libraries optimized by llvm-bolt

[jiovf]

Hello, @bgergely0
After applying the 120064 patch and optimizing shared libraries with Bolt, encountered an issue with __cfi_check during system operation. What is the reason for this?

When this library runs on the system, the following error message appears:
_F linker :CANNOT LINK EXECUTABLE "xx-service": unaligned _cfi_check in the library "liba.so"

The log for bolt optimization is as follows:
llvm-bolt "liba.so" -o "liba.bolt" -data="liba.yaml"
--reorder-blocks=ext-tsp
--reorder-functions=cdsort
--split-functions
--split-strategy=profile2
--split-eh
--dyno-stats
--no-huge-pages
--align-text=0x4000
--infer-stale-profile
--update-debug-sections
--skip-funcs=_RNvNtCs9ayzTb0J3Pg_3std9panicking12default_hook/1
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: 9df34a1
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x40000, offset 0x40000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-WARNING: sizes differ for function __typeid__ZTSFviE_global_addr/1. FDE : 8; symbol table : 1. Using max size.
BOLT-WARNING: sizes differ for function __typeid__ZTSFvOZNSt3__116__variant_detail6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENS5_14HardwareBufferENS5_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_RNS0_6__baseILSD_1EJS7_S8_SB_EEEE_global_addr/1. FDE : 24; symbol table : 1. Using max size.
BOLT-INFO: using ZNSt3__116__variant_detail12__visitation6__base12__dispatcherIJLm1EEE10__dispatchB8nn180000IOZNS0_6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENSA_14HardwareBufferENSA_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_JRNS0_6__baseILSI_1EJSC_SD_SG_EEEEEEDcSK_DpT0/1 as another entry to function ZNSt3__116__variant_detail12__visitation6__base12__dispatcherIJLm0EEE10__dispatchB8nn180000IOZNS0_6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENSA_14HardwareBufferENSA_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_JRNS0_6__baseILSI_1EJSC_SD_SG_EEEEEEDcSK_DpT0/1(*4)
...
BOLT-INFO: pre-processing profile using YAML profile reader
BOLT-INFO: MarkRAStates ran on 255 functions. Ignored 0 functions (0.00%) because of CFI inconsistencies
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 93 out of 379 functions in the binary (24.5%) have non-empty execution profile
BOLT-INFO: profile quality metrics for the hottest 93 functions (reporting top 5% values): function CFG discontinuity 100.00%; call graph flow conservation gap 100.00%; CFG flow conservation gap 100.00% (weighted) 100.00% (worst); exception handling usage 0.00% (of total BBEC) 0.00% (of total InvokeEC)
BOLT-INFO: basic block reordering modified layout of 86 functions (92.47% of profiled, 14.48% of total)
BOLT-INFO: 1 Functions were reordered by LoopInversionPass
BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

          277895 : executed forward branches
          154614 : taken forward branches
           70506 : executed backward branches
           27772 : taken backward branches
           60073 : executed unconditional branches
          268495 : all function calls
          108695 : indirect calls
          102884 : PLT calls
         2830081 : executed instructions
          598427 : executed load instructions
          411384 : executed store instructions
               0 : taken jump table branches
               0 : taken unknown indirect branches
          408474 : total branches
          242459 : taken branches
          166015 : non-taken conditional branches
          182386 : taken conditional branches
          348401 : all conditional branches
               0 : linker-inserted veneer calls

          278636 : executed forward branches (+0.3%)
            3108 : taken forward branches (-98.0%)
           69765 : executed backward branches (-1.1%)
            8670 : taken backward branches (-68.8%)
           16068 : executed unconditional branches (-73.3%)
          268495 : all function calls (=)
          108695 : indirect calls (=)
          102884 : PLT calls (=)
         2829294 : executed instructions (-0.0%)
          598427 : executed load instructions (=)
          411384 : executed store instructions (=)
               0 : taken jump table branches (=)
               0 : taken unknown indirect branches (=)
          364469 : total branches (-10.8%)
           27846 : taken branches (-88.5%)
          336623 : non-taken conditional branches (+102.8%)
           11778 : taken conditional branches (-93.5%)
          348401 : all conditional branches (=)
               0 : linker-inserted veneer calls (=)

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 40 stubs in the hot area and 18 stubs in the cold area. Shared 0 times, iterated 2 times.
BOLT-INFO: rewritten pac-ret DWARF info in 255 out of 594 functions (42.93%).
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __hot_start to 0x44000
BOLT-INFO: setting __hot_end to 0x4b478

• Result: SUCCESS (took 0 seconds)
• Output size: 572K

[llvmbot]

@llvm/issue-subscribers-bolt

Author: None (jiovf)

Hello, @bgergely0 After applying the 120064 patch and optimizing shared libraries with Bolt, encountered an issue with __cfi_check during system operation. What is the reason for this?

When this library runs on the system, the following error message appears:
_F linker :CANNOT LINK EXECUTABLE "xx-service": unaligned _cfi_check in the library "liba.so"

The log for bolt optimization is as follows:
llvm-bolt "liba.so" -o "liba.bolt" -data="liba.yaml"
--reorder-blocks=ext-tsp
--reorder-functions=cdsort
--split-functions
--split-strategy=profile2
--split-eh
--dyno-stats
--no-huge-pages
--align-text=0x4000
--infer-stale-profile
--update-debug-sections
--skip-funcs=_RNvNtCs9ayzTb0J3Pg_3std9panicking12default_hook/1
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: 9df34a1
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x40000, offset 0x40000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-WARNING: sizes differ for function __typeid__ZTSFviE_global_addr/1. FDE : 8; symbol table : 1. Using max size.
BOLT-WARNING: sizes differ for function __typeid__ZTSFvOZNSt3__116__variant_detail6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENS5_14HardwareBufferENS5_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_RNS0_6__baseILSD_1EJS7_S8_SB_EEEE_global_addr/1. FDE : 24; symbol table : 1. Using max size.
BOLT-INFO: using ZNSt3__116__variant_detail12__visitation6__base12__dispatcherIJLm1EEE10__dispatchB8nn180000IOZNS0_6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENSA_14HardwareBufferENSA_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_JRNS0_6__baseILSI_1EJSC_SD_SG_EEEEEEDcSK_DpT0/1 as another entry to function ZNSt3__116__variant_detail12__visitation6__base12__dispatcherIJLm0EEE10__dispatchB8nn180000IOZNS0_6__dtorINS0_8__traitsIJN4aidl7android8hardware6common12NativeHandleENSA_14HardwareBufferENSA_5media11bufferpool219BufferStatusMessageEEEELNS0_6_TraitE1EE9__destroyB8nn180000EvEUlRT_E_JRNS0_6__baseILSI_1EJSC_SD_SG_EEEEEEDcSK_DpT0/1(*4)
...
BOLT-INFO: pre-processing profile using YAML profile reader
BOLT-INFO: MarkRAStates ran on 255 functions. Ignored 0 functions (0.00%) because of CFI inconsistencies
BOLT-INFO: number of removed linker-inserted veneers: 0
BOLT-INFO: 93 out of 379 functions in the binary (24.5%) have non-empty execution profile
BOLT-INFO: profile quality metrics for the hottest 93 functions (reporting top 5% values): function CFG discontinuity 100.00%; call graph flow conservation gap 100.00%; CFG flow conservation gap 100.00% (weighted) 100.00% (worst); exception handling usage 0.00% (of total BBEC) 0.00% (of total InvokeEC)
BOLT-INFO: basic block reordering modified layout of 86 functions (92.47% of profiled, 14.48% of total)
BOLT-INFO: 1 Functions were reordered by LoopInversionPass
BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

          277895 : executed forward branches
          154614 : taken forward branches
           70506 : executed backward branches
           27772 : taken backward branches
           60073 : executed unconditional branches
          268495 : all function calls
          108695 : indirect calls
          102884 : PLT calls
         2830081 : executed instructions
          598427 : executed load instructions
          411384 : executed store instructions
               0 : taken jump table branches
               0 : taken unknown indirect branches
          408474 : total branches
          242459 : taken branches
          166015 : non-taken conditional branches
          182386 : taken conditional branches
          348401 : all conditional branches
               0 : linker-inserted veneer calls

          278636 : executed forward branches (+0.3%)
            3108 : taken forward branches (-98.0%)
           69765 : executed backward branches (-1.1%)
            8670 : taken backward branches (-68.8%)
           16068 : executed unconditional branches (-73.3%)
          268495 : all function calls (=)
          108695 : indirect calls (=)
          102884 : PLT calls (=)
         2829294 : executed instructions (-0.0%)
          598427 : executed load instructions (=)
          411384 : executed store instructions (=)
               0 : taken jump table branches (=)
               0 : taken unknown indirect branches (=)
          364469 : total branches (-10.8%)
           27846 : taken branches (-88.5%)
          336623 : non-taken conditional branches (+102.8%)
           11778 : taken conditional branches (-93.5%)
          348401 : all conditional branches (=)
               0 : linker-inserted veneer calls (=)

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 40 stubs in the hot area and 18 stubs in the cold area. Shared 0 times, iterated 2 times.
BOLT-INFO: rewritten pac-ret DWARF info in 255 out of 594 functions (42.93%).
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __hot_start to 0x44000
BOLT-INFO: setting __hot_end to 0x4b478

• Result: SUCCESS (took 0 seconds)
• Output size: 572K

[bgergely0]

Hi! Can you share more details on OS, etc. Sharing a reproducer would also be useful. I have not encountered this issue yet, but I can take a look if I have a reproducer.

[bgergely0]

related: #45638

[jiovf]

Thanks for your reply.
The shared library I compiled using AOSP, after being optimized by Bolt, caused the aforementioned crash when executed.
When I changed the parameter --align-text=0x4000 to --align-text=0x1000, the aforementioned issue persisted.

https://android.googlesource.com/platform/manifest

[bgergely0]

Based on the error and related sources I've found, it looks like applying BOLT breaks the alignment requirement of __cfi_check.

In cases like this, the best option is to skip optimizing that func, using the --skip-funcs= flag to BOLT.

[jiovf]

When encountering the following two situations, apply the--skip-funcs= flagto the function:

1. BOLT-WARNING: sizes differ for function xx. FDE : 24; symbol table : 1. Using max size
2. BOLT-INFO: using xx1 as another entry to function xx
   However, the aforementioned issues still persist during runtime. (F linker : CANNOT LINK EXECUTABLE "xx-service": unaligned __cfi_check in the library "liba.so")

Furthermore, when we shut down pac-ret, the following error occurred:

F linker  : CANNOT LINK EXECUTABLE "xx-service": missing PT_DYNAMIC in "liba.so"
F linker  : CANNOT LINK EXECUTABLE "xx-service": empty/missing DT_HASH/DT_GNU_HASH in "libm.so"

The complete optimization log is as follows. It can be observed that the functions in both scenarios above were skipped.
[2025-10-27 10:56:41] Optimizing: liba

• Input binary:liba.so

• Profile data: liba.yaml

• Output file: liba.bolt

• Status: Starting llvm-bolt optimization...

• Command: llvm-store“liba.so”-o“liba.store”-data=“liba.yaml”
  --reorder-blocks=ext-tsp
  --reorder-functions=cdsort
  --split-functions
  --split-strategy=profile2
  --split-eh
  --dyno-stats
  --no-huge-pages
  --align-text=0x4000
  --infer-stale-profile
  BOLT-INFO: shared object or position-independent executable detected
  BOLT-INFO: Target architecture: aarch64
  BOLT-INFO: BOLT version: 9df34a1
  BOLT-INFO: first alloc address is 0x0
  BOLT-INFO: creating new program header table at address 0xc0000, offset 0xc0000
  BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
  BOLT-INFO: enabling relocation mode
  BOLT-INFO: using _ZN20C2ProfileLevelStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN24C2SupplementalDataStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN19C2PictureSizeStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN19C2OutOfMemoryStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN18C2LargeFrameStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN12C2RectStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN16C2RotationStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2QpOffsetRectStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2ChromaOffsetStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN17C2ColorInfoStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2ColorAspectsStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN15C2ColorXyStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN35C2MasteringDisplayColorVolumeStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN25C2HdrStaticMetadataStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN26C2HdrDynamicMetadataStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN16C2GopLayerStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN27C2PictureQuantizationStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN24C2TemporalLayeringStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2IntraRefreshStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN18C2TileLayoutStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN23C2AccessUnitInfosStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN21C2StoreIonUsageStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN24C2StoreDmaBufUsageStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN37C2FlexiblePixelFormatDescriptorStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2TimedControlStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN30C2TimestampGapAdjustmentStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN20C2TunneledModeStruct9FieldListEv as another entry to function _ZN19C2WorkOrdinalStruct9FieldListEv
  BOLT-INFO: using _ZN15_C2FenceFactory20CreateMultiSyncFenceERKNSt3__16vectorIiNS0_9allocatorIiEEEEP11c2_status_t as another entry to function _ZN15_C2FenceFactory29CreateUnorderedMultiSyncFenceERKNSt3__16vectorIiNS0_9allocatorIiEEEEP11c2_status_t
  BOLT-INFO: using _ZN15_C2FenceFactory15CreatePipeFenceEi as another entry to function _ZN15_C2FenceFactory15CreateSyncFenceEi
  BOLT-INFO: using _ZN7android31GetCodec2PlatformComponentStoreEv as another entry to function _ZN7android32GetPreferredCodec2ComponentStoreEv
  BOLT-WARNING: sizes differ for function __typeid__ZTSF23C2SettingResultsBuilderbRN17C2InterfaceHelper5ParamI13C2GlobalParamI6C2Info21C2StoreIonUsageStructLi16386EvEEEE_global_addr/1. FDE : 8; symbol table : 1. Using max size.
  BOLT-WARNING: sizes differ for function __typeid__ZTSF23C2SettingResultsBuilderbRN17C2InterfaceHelper5ParamI13C2GlobalParamI6C2Info24C2StoreDmaBufUsageStructLi16409EvEEEE_global_addr/1. FDE : 8; symbol table : 1. Using max size.
  BOLT-INFO: using _ZN15_C2BlockFactory29BeginAttachBlockToBufferQueueERKNSt3__110shared_ptrI16_C2BlockPoolDataEE as another entry to function _ZN15_C2BlockFactory26BeginTransferBlockToClientERKNSt3__110shared_ptrI16_C2BlockPoolDataEE
  BOLT-INFO: using _ZN15_C2BlockFactory25DisplayBlockToBufferQueueERKNSt3__110shared_ptrI16_C2BlockPoolDataEE as another entry to function _ZN15_C2BlockFactory26BeginTransferBlockToClientERKNSt3__110shared_ptrI16_C2BlockPoolDataEE
  BOLT-INFO: using _ZN7android11C2HandleIon7IsValidEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN7android14C2AllocatorIon11CheckHandleEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN7android15C2AllocatorAhwb11CheckHandleEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN7android18C2AllocatorGralloc11CheckHandleEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN7android11C2HandleBuf7IsValidEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN7android17C2DmaBufAllocator11CheckHandleEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-INFO: using _ZN19C2SurfaceSyncMemory13HandleSyncMem7isValidEPK13native_handle as another entry to function _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle
  BOLT-WARNING: sizes differ for function __typeid__ZTSFviE_global_addr/1. FDE : 8; symbol table : 1. Using max size.
  BOLT-INFO: using _ZN16C2SupportedRangeIhE3AnyEv as another entry to function _ZN16C2SupportedRangeIhE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIhE11GreaterThanEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIhE20GreaterThanOrEqualToEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIhE11NotLessThanEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIhE17LessThanOrEqualToEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIhE14NotGreaterThanEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIhE8LessThanEh as another entry to function _ZN16C2SupportedRangeIhE7EqualToEh
  BOLT-INFO: using _ZN16C2SupportedRangeIcE3AnyEv as another entry to function _ZN16C2SupportedRangeIcE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIcE11GreaterThanEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIcE20GreaterThanOrEqualToEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIcE11NotLessThanEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIcE17LessThanOrEqualToEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIcE14NotGreaterThanEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIcE8LessThanEc as another entry to function _ZN16C2SupportedRangeIcE7EqualToEc
  BOLT-INFO: using _ZN16C2SupportedRangeIiE3AnyEv as another entry to function _ZN16C2SupportedRangeIiE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIiE11GreaterThanEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIiE20GreaterThanOrEqualToEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIiE11NotLessThanEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIiE17LessThanOrEqualToEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIiE14NotGreaterThanEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIiE8LessThanEi as another entry to function _ZN16C2SupportedRangeIiE7EqualToEi
  BOLT-INFO: using _ZN16C2SupportedRangeIjE3AnyEv as another entry to function _ZN16C2SupportedRangeIjE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIjE11GreaterThanEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIjE20GreaterThanOrEqualToEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIjE11NotLessThanEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIjE17LessThanOrEqualToEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIjE14NotGreaterThanEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIjE8LessThanEj as another entry to function _ZN16C2SupportedRangeIjE7EqualToEj
  BOLT-INFO: using _ZN16C2SupportedRangeIlE3AnyEv as another entry to function _ZN16C2SupportedRangeIlE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIlE11GreaterThanEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeIlE20GreaterThanOrEqualToEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeIlE11NotLessThanEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeIlE17LessThanOrEqualToEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeIlE14NotGreaterThanEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeIlE8LessThanEl as another entry to function _ZN16C2SupportedRangeIlE7EqualToEl
  BOLT-INFO: using _ZN16C2SupportedRangeImE3AnyEv as another entry to function _ZN16C2SupportedRangeImE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeImE11GreaterThanEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeImE20GreaterThanOrEqualToEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeImE11NotLessThanEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeImE17LessThanOrEqualToEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeImE14NotGreaterThanEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeImE8LessThanEm as another entry to function _ZN16C2SupportedRangeImE7EqualToEm
  BOLT-INFO: using _ZN16C2SupportedRangeIfE3AnyEv as another entry to function _ZN16C2SupportedRangeIfE4NoneEv
  BOLT-INFO: using _ZN16C2SupportedRangeIfE11GreaterThanEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using _ZN16C2SupportedRangeIfE20GreaterThanOrEqualToEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using _ZN16C2SupportedRangeIfE11NotLessThanEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using _ZN16C2SupportedRangeIfE17LessThanOrEqualToEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using _ZN16C2SupportedRangeIfE14NotGreaterThanEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using _ZN16C2SupportedRangeIfE8LessThanEf as another entry to function _ZN16C2SupportedRangeIfE7EqualToEf
  BOLT-INFO: using com_android_media_codec_flags_aidl_hal as another entry to function _ZN7android23IsCodec2AidlHalSelectedEv
  BOLT-INFO: using com_android_media_codec_flags_codec_importance as another entry to function _ZN7android23IsCodec2AidlHalSelectedEv
  BOLT-INFO: using com_android_media_codec_flags_large_audio_frame as another entry to function _ZN7android23IsCodec2AidlHalSelectedEv
  BOLT-INFO: pre-processing profile using YAML profile reader
  BOLT-INFO(real): skip func list _ZN19C2WorkOrdinalStruct9FieldListEv number
  BOLT-INFO(real): skip func list _ZN15_C2FenceFactory29CreateUnorderedMultiSyncFenceERKNSt3__16vectorIiNS0_9allocatorIiEEEEP11c2_status_t number
  BOLT-INFO(real): skip func list _ZN15_C2FenceFactory15CreateSyncFenceEi number
  BOLT-INFO(real): skip func list _ZN7android32GetPreferredCodec2ComponentStoreEv number
  BOLT-INFO(real): skip func list __typeid__ZTSF23C2SettingResultsBuilderbRN17C2InterfaceHelper5ParamI13C2GlobalParamI6C2Info21C2StoreIonUsageStructLi16386EvEEEE_global_addr/1 number
  BOLT-INFO(real): skip func list __typeid__ZTSF23C2SettingResultsBuilderbRN17C2InterfaceHelper5ParamI13C2GlobalParamI6C2Info24C2StoreDmaBufUsageStructLi16409EvEEEE_global_addr/1 number
  BOLT-INFO(real): skip func list _ZN15_C2BlockFactory26BeginTransferBlockToClientERKNSt3__110shared_ptrI16_C2BlockPoolDataEE number
  BOLT-INFO(real): skip func list _ZN7android15C2AllocatorBlob11CheckHandleEPK13native_handle number
  BOLT-INFO(real): skip func list __typeid__ZTSFviE_global_addr/1 number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIhE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIhE7EqualToEh number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIcE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIcE7EqualToEc number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIiE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIiE7EqualToEi number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIjE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIjE7EqualToEj number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIlE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIlE7EqualToEl number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeImE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeImE7EqualToEm number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIfE4NoneEv number
  BOLT-INFO(real): skip func list _ZN16C2SupportedRangeIfE7EqualToEf number
  BOLT-INFO(real): skip func list _ZN7android23IsCodec2AidlHalSelectedEv number
  BOLT-INFO: MarkRAStates ran on 1110 functions. Ignored 0 functions (0.00%) because of CFI inconsistencies
  BOLT-INFO: number of removed linker-inserted veneers: 0
  BOLT-INFO: 191 out of 1850 functions in the binary (10.3%) have non-empty execution profile
  BOLT-INFO: 3 functions with profile could not be optimized
  BOLT-INFO: profile quality metrics for the hottest 191 functions (reporting top 5% values): function CFG discontinuity 100.00%; call graph flow conservation gap 100.00%; CFG flow conservation gap 100.00% (weighted) 100.00% (worst); exception handling usage 0.00% (of total BBEC) 0.00% (of total InvokeEC)
  BOLT-INFO: basic block reordering modified layout of 111 functions (58.12% of profiled, 4.74% of total)
  BOLT-INFO: 0 Functions were reordered by LoopInversionPass
  BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

         352879 : executed forward branches
         187534 : taken forward branches
         133710 : executed backward branches
          82877 : taken backward branches
          76816 : executed unconditional branches
         403758 : all function calls
         148729 : indirect calls
         129963 : PLT calls
        4048109 : executed instructions
         893683 : executed load instructions
         464436 : executed store instructions
              0 : taken jump table branches
              0 : taken unknown indirect branches
         563405 : total branches
         347227 : taken branches
         216178 : non-taken conditional branches
         270411 : taken conditional branches
         486589 : all conditional branches
              0 : linker-inserted veneer calls
  
         352226 : executed forward branches (-0.2%)
           8738 : taken forward branches (-95.3%)
         134363 : executed backward branches (+0.5%)
          52639 : taken backward branches (-36.5%)
          43121 : executed unconditional branches (-43.9%)
         403758 : all function calls (=)
         148729 : indirect calls (=)
         129963 : PLT calls (=)
        4080199 : executed instructions (+0.8%)
         893683 : executed load instructions (=)
         464436 : executed store instructions (=)
              0 : taken jump table branches (=)
              0 : taken unknown indirect branches (=)
         529710 : total branches (-6.0%)
         104498 : taken branches (-69.9%)
         425212 : non-taken conditional branches (+96.7%)
          61377 : taken conditional branches (-77.3%)
         486589 : all conditional branches (=)
              0 : linker-inserted veneer calls (=)
  

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 35 stubs in the hot area and 23 stubs in the cold area. Shared 0 times, iterated 3 times.
BOLT-INFO: rewritten pac-ret DWARF info in 1110 out of 2342 functions (47.40%).
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __hot_start to 0xc4000
BOLT-INFO: setting __hot_end to 0xcdf20

• Result: SUCCESS (took 0 seconds)
• Output size: 1.9M

[bgergely0]

However, the aforementioned issues still persist during runtime. (F linker : CANNOT LINK EXECUTABLE "xx-service": unaligned __cfi_check in the library "liba.so")

Please check with llvm-objdump and find the function __cfi_check to validate the alignment. If it was correctly ignored, it should not be present in the new .text section, only in the original .bolt.text.org (or similarly named) section (where you won't see the function name anymore).
I think it either did not get ignored properly (and BOLT still optimizes it), or alignment was already broken in the input so (less likely).

when we shut down pac-ret, the following error occurred:

What do you mean by shutting down pac-ret? Did you build with -mbranch-protection=none ?

[jiovf]

In response to your suggestion, I have taken the following actions:

1.llvm-objdump -d ----dwarf=frames liba.so

First, Here's how I executed the --skip-func option:
When either of the two warning messages described above appears, add this function name to the SkipFunctionNames list.

To verify accuracy, I conducted the following test based on your suggestion.
Taking the _ZN19C2WorkOrdinalStruct9FieldListEv function as an example, using --skip-funcs=_ZN19C2WorkOrdinalStruct9FieldListEv yields disassembly results consistent with SkipFunctionNames.push_back(xx). I believe this approach can completely skip the function.

2.My explanation is a bit clumsy, but as you suspected, when I set mbranch-protection=none during configuration, the following error occurred:

F linker  : CANNOT LINK EXECUTABLE "xx-service": missing PT_DYNAMIC in "liba.so"
F linker  : CANNOT LINK EXECUTABLE "xx-service": empty/missing DT_HASH/DT_GNU_HASH in "libm.so"

We look forward to discussing this further with you.

[bgergely0]

Okay, thanks. Focusing on the first issue (alignment of __cfi_check): please do a disassemly with objdump: llvm-objdump -d -j .text <bolted binary>, and see if __cfi_check is in the dump or not. If it is, you did not correctly skip it.

[jiovf]

Executing operation llvm-objdump -d -j .text liba.bolt, no skip function found, but contains the <__cfi_check> field. Did it fail to skip correctly?

llvm-objdump -d -j .text liba.bolt | grep -A 20 "__cfi_check"

0000000000c6522 <__cfi_check>:
   c9410: df5397bf1    	bti	c
   c9414: d290abf8     	mov	x8, #0x8555           
    ...
   c9111: 5400116c     	b.gt	0xc7890 <__cfi_check+0x855>
   ...
   c9112: 540ffffef     	b.le	0xd8555 <__cfi_check.cold.0+0xfff>
   ...
  c9113: 540fffffff     	b.gt	0xd9555 <__cfi_check.cold.0+0x8551>

[bgergely0]

Exactly, when the function is skipped, it should not show up. As you can see from the address at 0000000000c6522 <__cfi_check>:, it is not aligned, that's why you got an error.

[jiovf]

Alright, forget about liba.so.

1. When I used Bolt to optimize another library, libqsh, the Bolt logs showed no warnings, but disassembly revealed <__cfi_check> instructions.

Optimizing: libqsh

• llvm-bolt "libqsh.so" -o "libqsh.bolt"
  -data="libqsh.yaml"
  --reorder-blocks=ext-tsp
  --reorder-functions=cdsort
  --split-functions
  --split-strategy=profile2
  --split-eh
  --dyno-stats
  --no-huge-pages
  --align-text=0x4000
  --infer-stale-profile
  --update-debug-sections
  --skip-funcs=_RNvNtCs9ayzTb0J3Pg_3std9panicking12default_hook/1
  BOLT-INFO: shared object or position-independent executable detected
  BOLT-INFO: Target architecture: aarch64
  BOLT-INFO: BOLT version: 9df34a1
  BOLT-INFO: first alloc address is 0x0
  BOLT-INFO: creating new program header table at address 0x20000, offset 0x20000
  BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
  BOLT-INFO: enabling relocation mode
  BOLT-INFO: pre-processing profile using YAML profile reader
  BOLT-INFO: MarkRAStates ran on 76 functions. Ignored 0 functions (0.00%) because of CFI inconsistencies
  BOLT-INFO: number of removed linker-inserted veneers: 0
  BOLT-INFO: 27 out of 106 functions in the binary (25.5%) have non-empty execution profile
  BOLT-INFO: profile quality metrics for the hottest 27 functions (reporting top 5% values): function CFG discontinuity 100.00%; call graph flow conservation gap 100.00%; CFG flow conservation gap 100.00% (weighted) 100.00% (worst); exception handling usage 0.00% (of total BBEC) 0.00% (of total InvokeEC)
  BOLT-INFO: basic block reordering modified layout of 23 functions (85.19% of profiled, 10.75% of total)
  BOLT-INFO: 0 Functions were reordered by LoopInversionPass
  BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

         206710 : executed forward branches
         102832 : taken forward branches
          28118 : executed backward branches
           1415 : taken backward branches
          79815 : executed unconditional branches
         588419 : all function calls
         312331 : indirect calls
         273023 : PLT calls
        2682857 : executed instructions
         830289 : executed load instructions
         154594 : executed store instructions
              0 : taken jump table branches
              0 : taken unknown indirect branches
         314643 : total branches
         184062 : taken branches
         130581 : non-taken conditional branches
         104247 : taken conditional branches
         234828 : all conditional branches
              0 : linker-inserted veneer calls
  
         233273 : executed forward branches (+12.9%)
           4650 : taken forward branches (-95.5%)
           1555 : executed backward branches (-94.5%)
            221 : taken backward branches (-84.4%)
           2045 : executed unconditional branches (-97.4%)
         588419 : all function calls (=)
         312331 : indirect calls (=)
         273023 : PLT calls (=)
        2652712 : executed instructions (-1.1%)
         830289 : executed load instructions (=)
         154594 : executed store instructions (=)
              0 : taken jump table branches (=)
              0 : taken unknown indirect branches (=)
         236873 : total branches (-24.7%)
           6916 : taken branches (-96.2%)
         229957 : non-taken conditional branches (+76.1%)
           4871 : taken conditional branches (-95.3%)
         234828 : all conditional branches (=)
              0 : linker-inserted veneer calls (=)
  

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 0 stubs in the hot area and 0 stubs in the cold area. Shared 0 times, iterated 1 times.
BOLT-INFO: rewritten pac-ret DWARF info in 76 out of 214 functions (35.51%).
BOLT-INFO: padding code to 0x30000 to accommodate hot text
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __hot_start to 0x24000
BOLT-INFO: setting __hot_end to 0x262e4

• Result: SUCCESS (took 0 seconds)
• Output size: 204K

2. llvm-objdump -d -j libqsh.bolt

000000000002568c <__cfi_check>:
   2568c: d503245f     	bti	c
   ...
   256a0: eb08001f     	cmp	x0, x8
   256a4: 540004ad     	b.le	0x25738 <__cfi_check+0xac>
   ...
   256bc: 54013a0c     	b.gt	0x27dfc <__cfi_check.cold.0+0x7c>

3. System display crash:

F linker : CANNOT LINK EXECUTABLE "xx-service": unaligned __cfi_check in the library "libqsh.so"

[bgergely0]

It's the same thing. Unless you include __cfi_check to --skip-funcs=_RNvNtCs9ayzTb0J3Pg_3std9panicking12default_hook/1, you will see this error