[AbstractCallSite] Handle Indirect Calls Properly

[Camsyn]

AbstractCallSite handles three types of calls (direct, indirect, and callback).

This patch fixes the handling of indirect calls in some methods, which incorrectly assumed that non-direct calls are always callback calls.

Moreover, this PR adds 2 unit tests for direct call type and indirect call type.

The aforementioned misassumption leads to the following problem:

---

Problem

When the underlying call is indirect, some APIs of AbstractCallSite behave unexpectedly.
E.g., AbstractCallSite::getCalledFunction() currently triggers an assertion failure, instead of returning nullptr as documented:

/// Return the function being called if this is a direct call, otherwise
/// return null (if it's an indirect call).
Function *getCalledFunction() const;

Actual unexpected assertion failure:

AbstractCallSite.h:197: int llvm::AbstractCallSite::getCallArgOperandNoForCallee() const: Assertion `isCallbackCall()' failed.

This is because AbstractCallSite mistakenly entered the branch that handles Callback Calls as its guard condition (!isDirectCall()) does not take into account the case of indirect calls

[llvmbot]

@llvm/pr-subscribers-llvm-ir

Author: Kunqiu Chen (Camsyn)

Changes

This patch fixes a bug in AbstractCallSite where indirect calls cause unexpected behavior and assertion failures.

Problem

When the underlying call is indirect, some APIs of AbstractCallSite behave unexpectedly.
E.g., AbstractCallSite::getCalledFunction() currently triggers an assertion failure, instead of returning nullptr as documented:

/// Return the function being called if this is a direct call, otherwise
/// return null (if it's an indirect call).
Function *getCalledFunction() const;

This violates the intended contract of AbstractCallSite and makes it unsafe to use as a general call-site abstraction.

Expected Behavior

AbstractCallSite is designed to provide a unified interface for representing all the THREE kinds of call sites — including direct calls, indirect calls, and callback calls.

It should gracefully handle indirect calls.

Why It Went Unnoticed

Currently, LLVM does not contain any in-tree use cases where AbstractCallSite wraps an indirect call, so this issue has not been triggered or tested before.

Motivation

• Improves robustness and aligns implementation with the documented behavior.
• Enables future and third-party extensions that use AbstractCallSite to represent arbitrary call sites.
• Ensures AbstractCallSite can safely wrap indirect calls without breaking internal invariants.

Fix

This pull request updates the logic in the AbstractCallSite class to more accurately distinguish between direct, indirect, and callback calls, and adds comprehensive unit tests for each call type.

The main change is that several methods now check if (!isCallbackCall()) rather than just if (isDirectCall()), ensuring correct behavior for indirect call sites.

---

Full diff: https://github.com/llvm/llvm-project/pull/163003.diff

2 Files Affected:

• (modified) llvm/include/llvm/IR/AbstractCallSite.h (+5-5)
• (modified) llvm/unittests/IR/AbstractCallSiteTest.cpp (+53)

diff --git a/llvm/include/llvm/IR/AbstractCallSite.h b/llvm/include/llvm/IR/AbstractCallSite.h
index 9e24ae7d1b431..f431e1d8a38ef 100644
--- a/llvm/include/llvm/IR/AbstractCallSite.h
+++ b/llvm/include/llvm/IR/AbstractCallSite.h
@@ -137,7 +137,7 @@ class AbstractCallSite {
 
   /// Return true if @p U is the use that defines the callee of this ACS.
   bool isCallee(const Use *U) const {
-    if (isDirectCall())
+    if (!isCallbackCall())
       return CB->isCallee(U);
 
     assert(!CI.ParameterEncoding.empty() &&
@@ -154,7 +154,7 @@ class AbstractCallSite {
 
   /// Return the number of parameters of the callee.
   unsigned getNumArgOperands() const {
-    if (isDirectCall())
+    if (!isCallbackCall())
       return CB->arg_size();
     // Subtract 1 for the callee encoding.
     return CI.ParameterEncoding.size() - 1;
@@ -169,7 +169,7 @@ class AbstractCallSite {
   /// Return the operand index of the underlying instruction associated with
   /// the function parameter number @p ArgNo or -1 if there is none.
   int getCallArgOperandNo(unsigned ArgNo) const {
-    if (isDirectCall())
+    if (!isCallbackCall())
       return ArgNo;
     // Add 1 for the callee encoding.
     return CI.ParameterEncoding[ArgNo + 1];
@@ -183,7 +183,7 @@ class AbstractCallSite {
   /// Return the operand of the underlying instruction associated with the
   /// function parameter number @p ArgNo or nullptr if there is none.
   Value *getCallArgOperand(unsigned ArgNo) const {
-    if (isDirectCall())
+    if (!isCallbackCall())
       return CB->getArgOperand(ArgNo);
     // Add 1 for the callee encoding.
     return CI.ParameterEncoding[ArgNo + 1] >= 0
@@ -210,7 +210,7 @@ class AbstractCallSite {
 
   /// Return the pointer to function that is being called.
   Value *getCalledOperand() const {
-    if (isDirectCall())
+    if (!isCallbackCall())
       return CB->getCalledOperand();
     return CB->getArgOperand(getCallArgOperandNoForCallee());
   }
diff --git a/llvm/unittests/IR/AbstractCallSiteTest.cpp b/llvm/unittests/IR/AbstractCallSiteTest.cpp
index ddb10911ad028..c30515a93b339 100644
--- a/llvm/unittests/IR/AbstractCallSiteTest.cpp
+++ b/llvm/unittests/IR/AbstractCallSiteTest.cpp
@@ -53,3 +53,56 @@ TEST(AbstractCallSite, CallbackCall) {
   EXPECT_TRUE(ACS.isCallee(CallbackUse));
   EXPECT_EQ(ACS.getCalledFunction(), Callback);
 }
+
+TEST(AbstractCallSite, DirectCall) {
+  LLVMContext C;
+
+  const char *IR = "declare void @bar()\n"
+                   "define void @foo() {\n"
+                   "  call void @bar()\n"
+                   "  ret void\n"
+                   "}\n";
+
+  std::unique_ptr<Module> M = parseIR(C, IR);
+  ASSERT_TRUE(M);
+
+  Function *Callee = M->getFunction("bar");
+  ASSERT_NE(Callee, nullptr);
+
+  const Use *DirectCallUse = Callee->getSingleUndroppableUse();
+  ASSERT_NE(DirectCallUse, nullptr);
+
+  AbstractCallSite ACS(DirectCallUse);
+  EXPECT_TRUE(ACS);
+  EXPECT_TRUE(ACS.isDirectCall());
+  EXPECT_TRUE(ACS.isCallee(DirectCallUse));
+  EXPECT_EQ(ACS.getCalledFunction(), Callee);
+}
+
+TEST(AbstractCallSite, IndirectCall) {
+  LLVMContext C;
+
+  const char *IR = "define void @foo(ptr %0) {\n"
+                   "  call void %0()\n"
+                   "  ret void\n"
+                   "}\n";
+
+  std::unique_ptr<Module> M = parseIR(C, IR);
+  ASSERT_TRUE(M);
+
+  Function *Fun = M->getFunction("foo");
+  ASSERT_NE(Fun, nullptr);
+
+  Argument *ArgAsCallee = Fun->getArg(0);
+  ASSERT_NE(ArgAsCallee, nullptr);
+
+  const Use *IndCallUse = ArgAsCallee->getSingleUndroppableUse();
+  ASSERT_NE(IndCallUse, nullptr);
+
+  AbstractCallSite ACS(IndCallUse);
+  EXPECT_TRUE(ACS);
+  EXPECT_TRUE(ACS.isIndirectCall());
+  EXPECT_TRUE(ACS.isCallee(IndCallUse));
+  EXPECT_EQ(ACS.getCalledFunction(), nullptr);
+  EXPECT_EQ(ACS.getCalledOperand(), ArgAsCallee);
+}

[nikic]

Kindly rewrite the PR description in your own words.

[Camsyn]

Kindly rewrite the PR description in your own words.

Sorry, I originally used an LLM to polish the PR description, thinking it would make the text more structured and easier to read.
It seems that was not the right approach. I’ve now rewritten the description entirely in my own words.

[Camsyn]

Friendly ping, @nikic 😀.

[nikic]

LGTM, though I'm still unhappy with the commit message. Please optimize for maximum information density, not for maximum PR description length.

A sufficient description for this change would be something like:

AbstractCallSite handles three types of call (direct, indirect and callback). Fix the handling of indirect calls in some methods, which incorrectly assumed that non-direct calls are always callback calls.

[nikic]

Also check the other methods like getNumArgOperands and getCallArgOperand?