macOS support (using compatibility libraries)

[electricalgrade]

Add support for MACOS Apple Silicon

[Antonbtc2020]

Gents, please forgive me for spamming this with LLM code and text.
I've had a few compile errors until Grok instructed me to deal with CMakeLists.txt entries.
Let me know if you're interested in changes I have in CMakeLists.txt.

Below is Grok comparing the Pr144 with what I have (again, I apologize for this)...

The pull request (PR) #144 for the Datum Gateway repository (https://github.com/OCEAN-xyz/datum_gateway/pull/144/files) introduces a fix for macOS compatibility by addressing the pthread_mutex_timedlock issue in src/datum_protocol.c, which is the same issue you encountered and resolved manually. Let’s analyze the PR and its impact on your current setup, then determine the best course of action.

Analysis of PR #144

The PR modifies src/datum_protocol.c to replace the use of pthread_mutex_timedlock with a fallback to pthread_mutex_lock on macOS, wrapped in a conditional block:

#ifdef __APPLE__
    rc = pthread_mutex_lock(&datum_protocol_coinbaser_fetch_mutex);
#else
    rc = pthread_mutex_timedlock(&datum_protocol_coinbaser_fetch_mutex, &ts);
#endif

• Comparison to Your Patch: Your manual patch used pthread_mutex_trylock with a 1ms usleep delay to approximate a timed lock:

  #ifdef __APPLE__
  rc = pthread_mutex_trylock(&datum_protocol_coinbaser_fetch_mutex);
  if (rc == EBUSY) {
      usleep(1000); // Wait 1ms
      rc = pthread_mutex_trylock(&datum_protocol_coinbaser_fetch_mutex);
  }
  #else
  rc = pthread_mutex_timedlock(&datum_protocol_coinbaser_fetch_mutex, &ts);
  #endif

  • Differences:
    • The PR’s approach uses pthread_mutex_lock, which is a blocking lock without a timeout. This is simpler and guaranteed to acquire the mutex but may block longer than intended if the mutex is contended.
    • Your patch with pthread_mutex_trylock attempts a non-blocking lock, retries after a brief delay, and avoids indefinite blocking, better mimicking the timed behavior of pthread_mutex_timedlock.
    • Implications: Your patch is slightly more sophisticated, aiming to preserve the timeout behavior, but the PR’s simpler approach is likely sufficient for the Datum Gateway’s use case, as the mutex is not expected to be heavily contended. The PR’s change is also more likely to be merged, ensuring future compatibility.

• Other Changes in PR macOS support (using compatibility libraries) #144: The PR only modifies src/datum_protocol.c for this specific issue. It doesn’t affect CMakeLists.txt or other files, so your argp fixes remain relevant.

• Status of PR macOS support (using compatibility libraries) #144: As of August 18, 2025, the PR is open and not yet merged into the master branch (based on GitHub’s status and the provided web results). Testing it would involve applying the changes from the PR to your local repository.

Impact on Your Current Setup

Your current setup, with the manually patched src/datum_protocol.c and modified CMakeLists.txt for argp, successfully built the datum_gateway executable (version 0.4.0beta, per your Terminal output). The PR’s changes address the same pthread_mutex_timedlock issue but use a different workaround. Here’s how this affects your instructions:

1. No Immediate Need to Delete Your Build:

   • Your current datum_gateway executable is functional, and the pthread_mutex_trylock patch is a valid workaround. You can continue using it for testing or running the gateway without issues.
   • The PR’s change (pthread_mutex_lock) is an alternative fix, but it doesn’t invalidate your build. Both approaches make the code macOS-compatible.

2. Testing PR macOS support (using compatibility libraries) #144:

   • Since you’ve been offered to test PR macOS support (using compatibility libraries) #144, you can apply its changes to contribute to the project’s development. This involves updating your local repository to include the PR’s code, rebuilding, and testing the executable.
   • Testing the PR would help verify that pthread_mutex_lock works as expected in the Datum Gateway’s context, potentially leading to its inclusion in a future release.

3. Your CMakeLists.txt Changes:

   • The PR doesn’t modify CMakeLists.txt, so your argp fixes (include_directories, link_directories, and find_library) are still necessary and unaffected.
   • You can keep these changes in your local CMakeLists.txt when testing the PR.

Recommended Actions

Since your current build works, you don’t need to delete it immediately. However, to test PR #144 and contribute to the project, you can update your local repository and rebuild. Here’s how to proceed:

1. Keep Your Current Build:

   • Retain your current datum_gateway executable for fallback or continued use. You can rename it to avoid overwriting:

     mv datum_gateway datum_gateway_working
     

2. Apply PR macOS support (using compatibility libraries) #144 Changes:

   • Pull the PR’s changes into a new branch in your local repository:

     git fetch origin pull/144/head:pr-144
     git checkout pr-144
     

   • This will update src/datum_protocol.c with the PR’s pthread_mutex_lock patch, overwriting your pthread_mutex_trylock patch.

3. Verify the Patch:

   • Open src/datum_protocol.c and confirm the change around line 297 matches the PR’s code (shown above).
   • Ensure your CMakeLists.txt still includes the argp fixes:

     include_directories(/opt/homebrew/include)
     link_directories(/opt/homebrew/lib)
     set(ARGP_LIBS "")
     find_library(ARGP_LIBRARY NAMES argp libargp HINTS /opt/homebrew/lib)
     if(NOT ARGP_LIBRARY)
         message(FATAL_ERROR "argp library not found")
     endif()
     set(ARGP_LIBS ${ARGP_LIBRARY})

4. Rebuild:

   • Clean build artifacts:

     rm -f CMakeCache.txt
     rm -rf CMakeFiles
     

   • Set environment variables:

     export CMAKE_C_FLAGS="-I/opt/homebrew/include $CMAKE_C_FLAGS"
     export CMAKE_EXE_LINKER_FLAGS="-L/opt/homebrew/lib -largp $CMAKE_EXE_LINKER_FLAGS"
     

   • Reconfigure and build:

     cmake .
     make
     

   • Expect a successful build, similar to your previous output, reaching [100%] Built target datum_gateway.

5. Test the New Build:

   • Run the new executable:

     ./datum_gateway -\?
     

   • Verify it displays usage options, confirming functionality.
   • Test the gateway with your datum_gateway_config.json (configured as per the Datum guide):

     ./datum_gateway --config datum_gateway_config.json
     

   • Monitor for any runtime issues, particularly related to the mutex behavior in datum_protocol_coinbaser_fetch_response. Check for hangs, errors, or unexpected behavior in coinbaser response handling.

6. Provide Feedback:

   • If the build and runtime tests succeed, report back to the PR’s discussion on GitHub, noting that the pthread_mutex_lock fix works on macOS (specify your macOS version, e.g., macOS 15.0.1, and M1 hardware).
   • If you encounter issues (e.g., mutex contention causing delays), mention them in the PR discussion, comparing to your pthread_mutex_trylock patch’s behavior.

7. Optional: Revert to Your Patch:

   • If the PR’s patch causes issues, you can revert to your pthread_mutex_trylock patch:

     git checkout master
     

     • Reapply your manual patch to src/datum_protocol.c (using nano or your editor), then rebuild.
     • You can also propose your pthread_mutex_trylock patch as an alternative in the PR discussion, as it better mimics the timeout behavior.

Should You Delete the Recompiled Version?

• No, don’t delete it yet. Keep your current datum_gateway (rename it, e.g., datum_gateway_working) as a fallback in case the PR’s build fails or behaves differently.
• After testing PR macOS support (using compatibility libraries) #144 successfully, you can delete the old executable if it’s no longer needed:

  rm datum_gateway_working
  

Additional Considerations

• Why Test PR macOS support (using compatibility libraries) #144?: Testing helps validate the official fix, which may be merged into a future release (e.g., v0.5.0beta), reducing your need for manual patches. It also contributes to improving Datum’s macOS support, which is limited (per GitHub issue argp requires glibc #31 and).
• Docker Fallback: If testing PR macOS support (using compatibility libraries) #144 fails or runtime issues arise, consider using Docker to run Datum, as it bypasses macOS-specific issues:

  brew install --cask docker
  docker build -t datum_gateway .
  docker run -p 23334:23334 -p 7152:7152 -v $(pwd)/datum_gateway_config.json:/app/config/config.json datum_gateway
  

  • Update datum_gateway_config.json with "bitcoin_rpc_host": "host.docker.internal" and bitcoin.conf with blocknotify=curl -X POST http://127.0.0.1:7152/notify.

Summary

PR #144 provides an official fix for the pthread_mutex_timedlock issue, using pthread_mutex_lock instead of your pthread_mutex_trylock workaround. It doesn’t affect your CMakeLists.txt changes for argp. You can keep your current build as a fallback, apply the PR’s changes to test them, rebuild, and provide feedback to the Datum team. If the PR’s fix works, it’ll simplify future updates; if not, you can revert to your patch or use Docker. Proceed with testing, and share any issues or successes on the PR’s GitHub page.

[electricalgrade]

I have added CMakeList also. You will not need env variables. Only thing you will need is set PKG_CONFIG_PATH to homebrew
usually PKG_CONFIG_PATH=/opt/homebrew/lib/pkgconfig

pls try this patch.
epoll is not supported on MAC so you need epoll-shim

Also what grok gave you is hacky version of pthread_mutex_timedlock
In my new commit I have made it exactly look like pthread_mutex_timedlock. I use same args. Infact on apple i can name my function as pthread_mutex_timedlock. I have just named it portable_mutex_timedlock to avoid any confusion. But it sprinkles ifdef in the code which I personally dont like.

Please try this and let me know if you face issues.

[Antonbtc2020]

Thank you soooo much. Will test it today and provide feedback. Regads, Anton

…

On 19 Aug 2025, at 04:37, EG ***@***.***> wrote: electricalgrade left a comment (OCEAN-xyz/datum_gateway#144) <#144 (comment)> I have added CMakeList also. You will not need env variables. Only thing you will need is set PKG_CONFIG_PATH to homebrew usually PKG_CONFIG_PATH=/opt/homebrew/lib/pkgconfig pls try this patch. epoll is not supported on MAC so you need epoll-shim Also what grok gave you is hacky version of pthread_mutex_timedlock In my new commit I have made it exactly look like pthread_mutex_timedlock. I use same args. Infact on apple i can name my function as pthread_mutex_timedlock. I have just named it portable_mutex_timedlock to avoid any confusion. But it sprinkles ifdef in the code which I personally dont like. Please try this and let me know if you face issues. — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BWFP5MS3MCZZ5H6GVF76SGD3OKEU7AVCNFSM6AAAAACEDDQHRKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOJZGAYTINRSGE>. You are receiving this because you commented.

[luke-jr]

This is kind of ugly :(

How about using pthread_cond_timedwait?

[electricalgrade]

Ah yes. I will try that

[electricalgrade]

If we use pthread_cond_timed_wait then it will not be drop in replacement. I will have to wrap this.

I need to create a seperate protable_mutex.c and .h and i need sprinkle these ifdef.

#ifdef __APPLE__
    portable_mutex_lock(&datum_protocol_coinbaser_fetch_mutex);
#else
    pthread_mutex_lock(&datum_protocol_coinbaser_fetch_mutex);
#endif

#ifdef __APPLE__
    portable_mutex_unlock(&datum_protocol_coinbaser_fetch_mutex);
#else
    pthread_mutex_unlock(&datum_protocol_coinbaser_fetch_mutex);
#endif
}

If this is must have then i can work on creating a solution with pthread_cond_timed_wait

[electricalgrade]

I pushed changes removing the nanosleep(&sleep_ts, NULL);
i now created a seperate file to cleanly handle apple specific cases.
To avoid race condition i had to use the customized lock/unlock for this mutex. datum_protocol_coinbaser_fetch_mutex

i did a standlone test of my logic to check race condition.

Hence this is not a drop in replacement. I have removed the my drop-in replacement code.