arch/sim: Add host socket and epoll APIs

[LingaoM]

NuttX sim renames many libc and POSIX symbols before the final link so
that NuttX applications do not accidentally bind to the host libc. This
is the right default for normal simulated applications, but it also makes
it difficult for selected sim-only code to intentionally access host
Linux interfaces such as Bluetooth HCI user channel sockets, Unix/TCP
sockets, epoll, and host file operations.

Add a small set of host-side wrappers for these interfaces. The new
helpers live in the sim host layer and are compiled as host objects, so
callers can explicitly opt in to host socket, epoll, and file operations
without bypassing the normal NuttX symbol namespace for the rest of the
application.

The socket helpers provide conversion between NuttX socket constants and
host socket constants, including support for Unix, IPv4, IPv6, Netlink,
and Bluetooth HCI socket addresses. Linux-only epoll support is kept in
a separate host epoll file and is only built for CONFIG_HOST_LINUX.

Extend the existing hostfs wrapper with additional file operations used
by sim applications, including pread, pwrite, fcntl, fsync, remove,
access, popen, fgets, and pclose.

Also split the socket ABI definitions that were previously local to
sim_hostusrsock.h into a shared sim_hostsock.h header. Definitions that
are specific to usrsock, such as message/control-message structures and
usrsock event flags, remain in sim_hostusrsock.h.

Testing:

Host:

• Ubuntu Linux x86_64
• GCC host toolchain

Board/configuration:

• sim:nsh

Commands:

./tools/configure.sh sim:nsh
make -j16

Result:

• Build completed successfully.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• arduino-mega2560
• esp32-devkitc
• hifive1-revb
• mirtoo
• qemu-armv8a
• qemu-intel64
• rx65n-rsk2mb
• s698pm-dkit
• stm32-nucleo-f103rb

[acassis]

@LingaoM please include proper testing. Just "build completed successfully" is not enough.

[acassis]

@LingaoM you wrote a good introduction to sim host layer and currently our Documentation about SIM is very poor: https://nuttx.apache.org/docs/latest/platforms/sim/sim/index.html
Please add this Summary as an Introduction to SIM, to let people understand how the SIM solves the functions names collision, etc

[linguini1]

Can you please tell us some use cases/test cases where you used these new host interfaces? If you have any logs that would be good to add.

[cederom]

Thank you @LingaoM, please read:

• https://nuttx.apache.org/docs/latest/contributing/index.html
• https://nuttx.apache.org/docs/latest/contributing/guide.html

[xiaoxiang781216]

All functions defined in hostfs.h aren't designed for application developer but consumed by https://github.com/apache/nuttx/tree/master/fs/hostfs.
I don't see that you modify the code under fs/hostfs/ to call new functions.
BTW, any application which call host_xxx function directly will be locked on sim board.

[xiaoxiang781216]

it's wrong to call the host socket api directly, since many functions (e.g. accept/connect/send/recv) will block the whole system, not just the current thread.

[xiaoxiang781216]

epoll_wait will block the whole OS, not just the current thread, which is definitely not you want behaviour.

[xiaoxiang781216]

NuttX sim renames many libc and POSIX symbols before the final link so that NuttX applications do not accidentally bind to the host libc. This is the right default for normal simulated applications, but it also makes it difficult for selected sim-only code to intentionally access host Linux interfaces such as Bluetooth HCI user channel sockets,

but we should use or improve: https://github.com/apache/nuttx/blob/master/arch/sim/src/sim/posix/sim_hosthcisocket.c
instead call host API directly since any block function will stop the whole OS.

Unix/TCP sockets,

should use usersock: https://github.com/apache/nuttx/blob/master/arch/sim/src/sim/posix/sim_hostusrsock.c

epoll, and

basically, you can't call Linux epoll api directly, since epoll_wait will block the whole OS, not just the calling thread.

host file operations.

should use: https://github.com/apache/nuttx/tree/master/fs/hostfs

Add a small set of host-side wrappers for these interfaces. The new helpers live in the sim host layer and are compiled as host objects, so callers can explicitly opt in to host socket, epoll, and file operations without bypassing the normal NuttX symbol namespace for the rest of the application.

it's wrong design to expose host api to application, since any application which call these functions will be locked on sim platform.
All arch specific feature must expose to the application through the standard interface, so the application can run on all targets.

Actually, all your modification already implements through the standard interface (hostfs/usersock/hci socket), why do you not use them?

The socket helpers provide conversion between NuttX socket constants and host socket constants, including support for Unix, IPv4, IPv6, Netlink, and Bluetooth HCI socket addresses.

All these functions can be achieved through usersock. IPv4/IPv6 is used daily in our project. Unix/Netlink/bluetooth need test.

Linux-only epoll support is kept in a separate host epoll file and is only built for CONFIG_HOST_LINUX.

Extend the existing hostfs wrapper with additional file operations used by sim applications, including pread, pwrite, fcntl, fsync, remove, access, popen, fgets, and pclose.

Also split the socket ABI definitions that were previously local to sim_hostusrsock.h into a shared sim_hostsock.h header. Definitions that are specific to usrsock, such as message/control-message structures and usrsock event flags, remain in sim_hostusrsock.h.

again, you can't call host socket api directly, otherwise the blocking call will block the whole system.

[LingaoM]

@xiaoxiang781216 Thanks for the review. I agree with the general design rule here: sim applications should not call blocking host APIs directly, because a blocking host call can block the whole simulated OS. I also agree that the long-term upstream direction should be standard NuttX interfaces, not application-visible host APIs.

The reason I did not simply switch everything to the existing standard backends is that the current implementations do not fully cover this our use case yet.

1. usersock is not currently enough for our transport set.

   The current sim usersock backend only accepts NUTTX_PF_INET; other domains return -EINVAL:
   sim_hostusrsock.c.

   Our test transport needs more than IPv4/TCP. The local user channel wrapper currently handles Bluetooth HCI, Unix domain sockets, and IPv4

   But Unix-domain and HCI-style transports are not covered by the current usersock implementation yet.

2. sim_hosthcisocket.c covers a different HCI use case.

   The existing host HCI helper opens a host Bluetooth HCI user channel with socket(PF_BLUETOOTH, SOCK_RAW | SOCK_CLOEXEC | SOCK_NONBLOCK, BTPROTO_HCI) and binds it by dev_idx/
   HCI_CHANNEL_USER:
   sim_hosthcisocket.c.

   In our integration, that is a Unix-socket HCI transport, not only a host hciX user-channel transport. I think the better upstream direction is to extend the existing sim HCI/usersock infrastructure to support this transport, rather than keeping a separate application-visible host socket API.

3. hostfs is the right standard interface for files, but file locking is a real gap for this workload.

   UnQLite uses POSIX advisory locks through fcntl(F_GETLK) and fcntl(F_SETLK):
   In NuttX VFS, F_SETLK is converted into the private ioctl FIOC_SETLK:

   But hostfs currently forwards ioctl commands to host_ioctl() and only special-cases FIOC_FILEPATH; it does not translate FIOC_SETLK/FIOC_GETLK/FIOC_SETLKW back into host fcntl()

   So I agree the application should use hostfs, but hostfs needs to support the lock ioctl path correctly for UnQLite-style databases.

4. For epoll, I agree raw blocking host epoll_wait() should not be exposed to applications.

   Our local adapter avoids blocking by calling host epoll_wait() with timeout 0 and then sleeping/yielding through the OS layer. However, I agree this is not the ideal upstream interface. The better upstream design is either to use NuttX poll/select/epoll on NuttX descriptors, or to keep any host polling strictly inside a sim backend that never blocks the simulated OS.

[LingaoM]

One more point: this was not originally intended as a NuttX core change. Our first choice was to keep this entirely inside the application/porting layer. However, in sim builds NuttX intentionally renames libc/POSIX symbols before the final link, so application code cannot reliably opt in to selected host Linux APIs by itself. That restriction is real and is defined by the sim symbol renaming mechanism.

If there is an accepted way for sim-only application code to explicitly opt in to host Linux APIs without bypassing NuttX’s normal interfaces, I am happy to move this PR out of the core changes or close that.

Actually, I also think this may be the cleaner direction: provide an explicit sim-only opt-in mechanism for applications that intentionally need host Linux APIs. The default NuttX behavior should remain unchanged: applications should use standard NuttX interfaces, and libc/POSIX symbols should continue to be renamed to avoid accidental host binding.

With that mechanism, we could keep these host-Linux shims in the application/porting layer and avoid adding workload-specific APIs to the NuttX core repository. The NuttX core would only provide the opt-in path; the app would decide whether to use it.

[xiaoxiang781216]

@xiaoxiang781216 Thanks for the review. I agree with the general design rule here: sim applications should not call blocking host APIs directly, because a blocking host call can block the whole simulated OS. I also agree that the long-term upstream direction should be standard NuttX interfaces, not application-visible host APIs.

The reason I did not simply switch everything to the existing standard backends is that the current implementations do not fully cover this our use case yet.

1. usersock is not currently enough for our transport set.
   The current sim usersock backend only accepts NUTTX_PF_INET; other domains return -EINVAL:
   sim_hostusrsock.c.
   Our test transport needs more than IPv4/TCP. The local user channel wrapper currently handles Bluetooth HCI, Unix domain sockets, and IPv4
   But Unix-domain and HCI-style transports are not covered by the current usersock implementation yet.

it's better to improve usrsock infrastructure to allow forward more socket type to host.

2. sim_hosthcisocket.c covers a different HCI use case.
   The existing host HCI helper opens a host Bluetooth HCI user channel with socket(PF_BLUETOOTH, SOCK_RAW | SOCK_CLOEXEC | SOCK_NONBLOCK, BTPROTO_HCI) and binds it by dev_idx/
   HCI_CHANNEL_USER:
   sim_hosthcisocket.c.
   In our integration, that is a Unix-socket HCI transport, not only a host hciX user-channel transport. I think the better upstream direction is to extend the existing sim HCI/usersock infrastructure to support this transport, rather than keeping a separate application-visible host socket API.

Yes, I think so.

3. hostfs is the right standard interface for files, but file locking is a real gap for this workload.
   UnQLite uses POSIX advisory locks through fcntl(F_GETLK) and fcntl(F_SETLK):
   In NuttX VFS, F_SETLK is converted into the private ioctl FIOC_SETLK:
   But hostfs currently forwards ioctl commands to host_ioctl() and only special-cases FIOC_FILEPATH; it does not translate FIOC_SETLK/FIOC_GETLK/FIOC_SETLKW back into host fcntl()

Why not use NuttX file lock support? but want forward to host fs.
Do you want the file lock work between NuttX simulator and the host apps?

BTW, it plans to add file lock/unlock into mount_operation to support the hostfs/virtiofs/nfs usage.

So I agree the application should use hostfs, but hostfs needs to support the lock ioctl path correctly for UnQLite-style databases.
4. For epoll, I agree raw blocking host epoll_wait() should not be exposed to applications.
Our local adapter avoids blocking by calling host epoll_wait() with timeout 0 and then sleeping/yielding through the OS layer. However, I agree this is not the ideal upstream interface. The better upstream design is either to use NuttX poll/select/epoll on NuttX descriptors, or to keep any host polling strictly inside a sim backend that never blocks the simulated OS.