gfal2 availability in Homebrew

[gabrielefronze]

On macOS, gfal2 and the other related packages should be install-able via the procedure shown here: https://github.com/cern-fts/homebrew-dmc

Unfortunately the whole install process fails due to missing formula for globus-toolkit in home-brew.
The legacy formula for GT is still relative to the old Globus website and repository, now 404.

Would it be possible to fix this install process to allow gfal2 usage on macOS?
This is highly required for tools like rucio which are getting widely used.
I am available to handle this task.

[mpatrascoiu]

Hello,

I don't think I'll be able to have a look at it very soon.
However, if you're willing to take the task, that sounds great!

Before you start, have a look at this pending issue regarding davix, one of the gfal2 dependencies:
cern-fts/homebrew-dmc#5

It might also affect this issue.

Cheers,
Mihai

[gabrielefronze]

Hi Mihai,

I think I can get some useful information from the issue you posted, thank you!
Basically I'd like to understand what are the functionalities of globus-toolkit used by gfal2: I need to provide them with a different toolset (read: set of packages).
Any advice is greatly appreciated :)

Cheers,
Gabriele

[mpatrascoiu]

Hello Gabriele,

The globus-toolkit provides necessary libraries and headers that Gfal2 uses for the gridFTP and SRM protocols.
With some CMake tinkering, it may be possible to disable those plugins, but you'd lose important protocols.

Perhaps there's a way to get them from the repo directly:
https://github.com/globus/globus-toolkit

I'm not familiar with the package distribution on MacOS.

Cheers,
Mihai

[gabrielefronze]

Hello Mihai,

I managed to work a bit on this.
Right now I had to disable quite a few protocols via CMakeLists:

#add_subdirectory (dcap)
add_subdirectory (file)
add_subdirectory (gridftp)
#add_subdirectory (http)
#add_subdirectory (lfc)
#add_subdirectory (rfio)
add_subdirectory (sftp)
#add_subdirectory (srm)
add_subdirectory (xrootd)
add_subdirectory (mock)

This is due to lack of many dependencies for such protocols via Home-brew (cap, davix_copy, LFC, RFIO and SRM).
Right now I am facing the errors directly related to the Globus Toolkit:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
GLOBUS_COMMON_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_FTP_CLIENT_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_FTP_CONTROL_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_GASS_COPY_INCLUDE_DIRS (ADVANCED)
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
   used as include directory in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_GASS_COPY_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_GSSAPI_GSI_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp
GLOBUS_GSS_ASSIST_LIBRARIES (ADVANCED)
    linked by target "plugin_gridftp" in directory /Users/gabrielegaetanofronze/gitstuff/gfal2/src/plugins/gridftp

Such errors do disappear if I comment out GridFTP as well.
Do you see any option to circumvent this manually adding the necessary dependencies getting rid of the globus-toolkit one?

Cheers,

Gabriele

[mpatrascoiu]

Hello Gabriele,

Unfortunately, I'm not very familiar with MacOS packaging.
The globus toolkit dependencies are needed for some very essential protocols, such as GridFTP and SRM.
Gfal2 won't be of much use without those protocols.

The problem is globus-toolkit is no longer maintained, as Globus discontinued support and now offer their cloud product.
However, the globus toolkit repository seems to still be active.
Perhaps there is a way to install/build the package from there?
If those dependencies are met, Gfal2 should also build.

Hope it helps.
Mihai

[ShamrockLee]

Some news about Mac packaging of gfal2:

I made it to package gfal2, gfal2-python and gfal2-util with Nix and Nixpkgs tooling. It's now on its way to Nixpkgs:

NixOS/nixpkgs#200100

The term Darwin refers to the the kernel of Mac OS, and is basically another way to say "Mac" in Nixpkgs scenario.

[gwarf]

The problem is globus-toolkit is no longer maintained

Sorry for commenting on that old message, and you may likely be already aware, but I would like to point to the work from the Grid Community Forum, and the work on the fork of the Globus Toolkit.
See https://gridcf.org/ and https://github.com/gridcf/gct/.

[ShamrockLee]

I would like to point to the work from the Grid Community Forum, and the work on the fork of the Globus Toolkit.\nSee https://gridcf.org/ and https://github.com/gridcf/gct/.

That direction is reasonable IMO.

As Grid Community Toolkit is huge and has a bunch of dependencies awaiting packaging, I just disable related plugins with the corresponding CMake flags -DPLUGIN_<NAME>=FALSE.

[ShamrockLee]

Hi! I'm packaging gfal2, gfal2-python and gfal2-util for Nixpkgs, the package repository of the cross-platform package manager Nix. The work can be found in PR NixOS/nixpkgs#200100.

I don't have MacOS device on hand. The CI build result of gfal2 with all plugin disabled shows a great amount of compilation error on both x86_64-based (x86_64-darwin) and arm-based (aarch64-darwin) MacOS platforms.

Does this project support MacOS? Are there some MacOS-specific dependencies or configuration needed to make it build?

[mpatrascoiu]

Hello,

To me, it looks like the errors, thus far, are all originating from the GTest library.

However, gfal2 is part of the DMC stack ([Grid] Data Management Clients) and it requires several lower-level dependencies. We have no plans to support MacOS platform.

[ShamrockLee]

@mpatrascoiu Thanks a lot for helping out!

BTW, I recently made gfal2, python3Packages.gfal2-python and gfal2-util into Nixpkgs, and they are now available on macOS through the Nix package manager. macOS functionality tests targeting gfal-ls and gfal-copy from root:// sources without a proxy are tested on the CI.