System calls via FFI

[sanko]

I've had this idea for a while now and might put effort into it next. I understand that this would be a massive task but the first 30% of it is already accomplished by infix existing in its current state.

Here's my proposal:

The design of this will be based on the same principles as the rest of infix: one-time setup for performance, declarative signature string, and clean separation between the generic, user-facing API and the messy platform-specific backends.

The Core API: A Familiar, Handle-Based Pattern

Instead of a single, syscall() function, we would mirror the trampoline pattern by creating a handle. This allows infix to perform the expensive lookup and JIT-compilation once.

/**
 * @brief An opaque handle to a JIT-compiled system call stub.
 */
typedef struct infix_syscall_t infix_syscall_t;

/**
 * @brief Creates a handle to a specific system call.
 *
 * This function looks up the syscall by its platform-specific name, parses the
 * signature, and generates a JIT stub to invoke it. Like a bound trampoline.
 *
 * @param[out] out_handle Receives the created handle.
 * @param[in] syscall_specifier A string identifying the syscall, using a
 *            platform namespace (e.g., "linux:write", "win:NtCreateFile").
 * @param[in] signature The infix signature of the syscall's arguments and return type.
 * @param[in] registry An optional registry for any named types used in the signature.
 * @return INFIX_SUCCESS on success.
 */
infix_status infix_syscall_create(
    infix_syscall_t ** out_handle,
    const char * syscall_specifier,
    const char * signature,
    infix_registry_t * registry
);

/**
 * @brief Invokes the system call.
 *
 * This function is the syscall version of of `infix_cif_func`. It executes
 * the JIT-compiled stub.
 *
 * @param handle The handle created by infix_syscall_create.
 * @param out_return_value A pointer to a buffer to receive the raw return value
 *        (e.g., an intptr_t). This value may indicate an error, which must be
 *        interpreted according to the OS's conventions (e.g., negative value for errno).
 * @param args An array of pointers to the argument values.
 */
void infix_syscall_call( infix_syscall_t * handle, void * out_return_value, void ** args );

/**
 * @brief Destroys a syscall handle and frees its resources.
 */
void infix_syscall_destroy( infix_syscall_t * handle );

A Portability 'Solution': The "Syscall Specifier"

The biggest challenge is naming. write means different things on different OSes so my proposed solution is a namespaced string: "os_name::syscall_name".

• "linux::write": Refers to the write syscall on any Linux system.
• "win::NtWriteFile": Refers to the NtWriteFile syscall on Windows.
• "freebsd::write": Refers to the write syscall on FreeBSD.
• "darwin::write": Refers to the write syscall on macOS/Darwin.

This approach acknowledges that syscalls are not truly portable at a semantic level and requires the developer to know their target OS. I might even need to add a version number to this somehow. Either way, this naming system makes the backend implementation clean and manageable, as each namespace can have its own simple name-to-number lookup table.

Example Usage: Writing to stdout

Here's how you'd call write(1, "hello\n", 6) on Linux:

#include <infix/infix.h>
#include <stdio.h>
#include <unistd.h> // For ssize_t

void portable_syscall_example_linux() {
    infix_syscall_t * handle = NULL;
    const char* specifier = "linux::write";
    // The signature for ssize_t write(int fd, const void *buf, size_t count);
    const char * signature = "(int, *void, uint64) -> sint64"; // Same signature language as infix...

    // Create the handle once.
    infix_status status = infix_syscall_create(&handle, specifier, signature, NULL);
    if (status != INFIX_SUCCESS)
        // Handle error...
        return;
  
    // Prepare arguments for a specific call.
    int fd = 1; // stdout
    const char * message = "Hello from a direct syscall!\n";
    uint64_t count = 29;
    void * args[] = { &fd, &message, &count };
    int64_t return_value;

    // Call the syscall. This is the fast part.
    infix_syscall_call( handle, &return_value, args );

    if (return_value < 0)
        printf("Syscall failed with errno: %lld\n", (long long) 0-return_value);
    else 
        printf("Syscall returned %lld (bytes written).\n", (long long)return_value);
    
    infix_syscall_destroy(handle);
}

Internals

I'll probably add a new v-table specific to system calls. Probably something like this in infix_internals.h:

typedef struct {
    // Looks up a name like "write" and returns its number for this OS.
    long (*lookup_syscall_number)(const char * name);
    // The ABI for syscalls (which registers to use for number and args).
    // This could even reuse parts of the main ABI spec.
    infix_abi_spec * syscall_abi;
    // The specific instruction to emit (e.g., the bytes for `syscall` or `svc`).
    void (* generate_syscall_instruction)(code_buffer * buf);
} infix_syscall_spec;

When infix_syscall_create is called (assume linux)...

• ...it parses the specifier (e.g., "linux::write").

• ...it selects the correct infix_syscall_spec for "linux".

• ...it calls spec->lookup_syscall_number("write") to get the number (e.g., 1).

• ...it generates a trampoline that:

  • Moves the syscall number 1 into the correct register (e.g., rax).
  • Moves the user's arguments (fd, message, count) into the correct argument registers (rdi, rsi, rdx).
  • Calls spec->generate_syscall_instruction() to emit the syscall instruction.
  • Handles the return value from rax.

---

Before I write any actual code, I've tried to piece together a few solid resources for finding the system call tables and conventions for each major supported platform.

---

Linux

Linux has a stable and well-documented syscall ABI for each architecture. The syscall number is placed in a specific register (rax on x86-64, x8 on AArch64), and then the syscall (or svc) instruction is executed.

Sources:

• https://filippo.io/linux-syscall-table/ is a very easy-to-use reference. It has as a dropdown to switch between architectures (x86-64, aarch64, etc.).
• https://syscalls64.paolostivanin.com/ also includes information on the registers used for each argument.
• syscalls(2)/man syscall/man syscalls man pages provide a complete list of all system calls and some examples.
• The kernel's unistd.h header for each architecture would be the definitive list to go by but might be a little dense.

---

Windows (Native API)

Windows does not have a stable, numbered syscall interface in the same way as Linux. Instead, the lowest-level supported interface is the Native API, which consists of functions exported by ntdll.dll (e.g., NtCreateFile, NtWriteFile). While these functions ultimately use a syscall or sysenter instruction, the numbers can change between Windows versions and even service packs. I'll need to enhance the OS detection in infix_config.h considerably. The only 'stable' interface is the function in ntdll.dll.

Sources:

• https://j00ru.vexillium.org/syscalls/nt/64/ is awesome. It documents the syscall numbers for Windows versions going back to Windows XP SP1. I probably won't interact with these directly but it's good to keep this link around.
• https://www.geoffchappell.com/studies/windows/win32/ntdll/index.htm?tx=8 is a great resource for otherwise undocumented ntdll.dll functions.
• As a last (or first) resort, I'll need to inspect the exported functions of ntdll.dll.
• https://github.com/reactos/reactos/blob/master/dll/ntdll/def/ntdll.spec from ReactOS's research.

---

macOS (XNU Kernel)

macOS uses a combination of POSIX-style syscalls and lower-level "Mach traps". It's based on FreeBSD and the numbers are defined in the kernel source code which is open source: https://github.com/apple-oss-distributions/xnu.

• The canonical list of syscalls, their numbers, and argument counts is in the syscalls.master file within the XNU kernel source code and can be found here: https://github.com/apple-oss-distributions/xnu/blob/main/bsd/kern/syscalls.master

---

FreeBSD

FreeBSD also uses a central file in the kernel source to define its syscalls: https://cgit.freebsd.org/src/tree/sys/kern/syscalls.master