CUDA cuFFT compilation error and suggested approach

[telegraphic]

Hello, I'm trying to use meson to compile a library that needs to be linked with a static library from CUDA, libcufft_static.a (usually located in /usr/local/cuda/lib64).

In short, I need to:

• add a flag -dc when generating a object file fft_kernels.o (only needed for this one object, not needed for other objects in the source code)
• I also need to make sure the -G flag is not passed to nvcc for these commands, otherwise it fail with strange PTX errors.
• Run a specific command using nvprune on the libcufft_static.a file
• generate a _device_link.o file from fft_kernels.o, with a -dlinkargument and-lcufft_static_pruned` argument
• Create a shared library from all .o files and _device_link.o

How would I do this with meson?

The existing Makefile has the following:

ifeq ($(HAVE_CUDA),1)
# TODO: Need to deal with 32/64 detection here
LIBCUFFT_STATIC = $(CUDA_LIBDIR64)/libcufft_static.a
# All PTX archs included in the lib (typically only one)
CUFFT_PTX_ARCHS := $(shell @CUOBJDUMP@ --list-ptx $(LIBCUFFT_STATIC) | grep -Po "sm_[0-9]{2}" | cut -d_ -f2 | sort | uniq)
# Latest PTX arch included in the lib
CUFFT_PTX_LATEST_ARCH := $(shell echo $(CUFFT_PTX_ARCHS) | rev | cut -d' ' -f1 | rev)
CUFFT_STATIC_GENCODE = -gencode arch=compute_$(CUFFT_PTX_LATEST_ARCH),\"code=compute_$(CUFFT_PTX_LATEST_ARCH)\"
libcufft_static_pruned.a: $(LIBCUFFT_STATIC) Makefile
	# We prune out all archs except those to be included in libbifrost *and*
	#   the latest PTX arch included in libcufft_static.
	#   E.g., We may have GPU_ARCHS="35 61" but libcufft_static might only
	#     include sm_60 and compute_60, so we need to keep compute_60 in order
	#     to support sm_61.
	@NVPRUNE@ -o $@ $(NVCC_GENCODE) $(CUFFT_STATIC_GENCODE) $<
fft_kernels.o: fft_kernels.cu fft_kernels.h Makefile
	# Note: This needs to be compiled with "-dc" to make CUFFT callbacks work
	$(NVCC) $(NVCCFLAGS) $(CPPFLAGS) -Xcompiler "$(GCCFLAGS)" -dc $(OUTPUT_OPTION) $<
_cuda_device_link.o: Makefile fft_kernels.o libcufft_static_pruned.a
	@echo "Linking _cuda_device_link.o"
	# TODO: "nvcc -dlink ..." does not error or warn when a -lblah is not found
	@ls ./libcufft_static_pruned.a > /dev/null
	$(NVCC) -dlink -o $@ $(NVCCFLAGS) fft_kernels.o -L. -lcufft_static_pruned
CUDA_DEVICE_LINK_OBJ = _cuda_device_link.o
else
CUDA_DEVICE_LINK_OBJ =
endif

Which runs something like:

/usr/local/cuda/bin/nvcc -std=c++11 -O3 -Xcompiler "-Wall" -default-stream per-thread -DBF_CUDA_ENABLED=1 -Xcompiler "-march=native" -Xcompiler " -fPIC -DPIC"  -gencode arch=compute_60,\"code=sm_60\"  -gencode arch=compute_61,\"code=sm_61\" -gencode arch=compute_61,\"code=compute_61\" -DBF_CUDA_ENABLED=1 -I. -I. -I/usr/local/cuda/include -Xcompiler "-fmessage-length=80 " -dc -o fft_kernels.o fft_kernels.cu

/usr/local/cuda/bin/nvprune -o libcufft_static_pruned.a  -gencode arch=compute_60,\"code=sm_60\"  -gencode arch=compute_61,\"code=sm_61\" -gencode arch=compute_61,\"code=compute_61\" -gencode arch=compute_80,\"code=compute_80\" /usr/local/cuda/lib64/libcufft_static.a

/usr/local/cuda/bin/nvcc -dlink -o _cuda_device_link.o -std=c++11 -O3 -Xcompiler "-Wall" -default-stream per-thread -DBF_CUDA_ENABLED=1 -Xcompiler "-march=native" -Xcompiler " -fPIC -DPIC"  -gencode arch=compute_60,\"code=sm_60\"  -gencode arch=compute_61,\"code=sm_61\" -gencode arch=compute_61,\"code=compute_61\" fft_kernels.o -L. -lcufft_static_pruned

Then this _cuda_device_link.o is used when creating the shared library:

g++ -std=c++14 -shared -Wl,--version-script=libbifrost.version,-soname,libbifrost.so.0 -o ../lib/libbifrost.so.0.10 common.o list_of_other_o_files_here.o _cuda_device_link.o -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib  -lcuda -lcudart -lnvrtc -L. -lcufft_static_pruned -lculibos -lnvToolsExt

Any guidance would be most appreciated!

So far my approach to create libcufft_static with meson is:

    # set cuda_flags by hand until I figure out a better way
    r = run_command('nvprune', 
                '/usr/local/cuda/lib64/libcufft_static.a',
                cuda_flags, 
                '-o', meson.build_root()+'/'+'libcufft_static_pruned.a'
                )
    cufft_pruned = cc.find_library('cufft_static_pruned', dirs: meson.build_root())

[telegraphic]

PS: The code is called bifrost, my meson branch is at https://github.com/bf-plugins/bifrost/tree/meson-build

[eli-schwartz]

For starters, I think your run_command should probably be a custom_target. Note that you can specify a custom_target output as a link_with: value, specifically to handle cases like this where libraries are created using specialized rules that are too complicated to handle with library().

• add a flag -dc when generating a object file fft_kernels.o (only needed for this one object, not needed for other objects in the source code)

Is it harmful to specify it for all of them? Meson doesn't let you define attributes on individual object files, but you could define it in an object library such as static_library('fft_kernels_stlib', 'fft_kernels.c', cuda_args: [...], build_by_default: false).extract_all_objects(recursive: false)

• I also need to make sure the -G flag is not passed to nvcc for these commands, otherwise it fail with strange PTX errors.

Meson will automatically define this flag if the buildtype is debug. Even if the project is configured with debug, you can force it to something else for this one target. Use override_options: ['buildtype=plain'].

• generate a _device_link.o file from fft_kernels.o, with a -dlinkargument and-lcufft_static_pruned` argument

Pass the extract_all_objects() opaque return value representing the compiled fft_kernels.o, as the input parameter of a custom_target. List that in the command: kwarg using '@INPUT@' style replacement.

Create a shared library from all .o files and _device_link.o

shared_library('foobar', objects: [list, of, extracted_objects])

[telegraphic]

Thanks @eli-schwartz , I'll give this a go 🙏

[telegraphic]

Here's my working approach using the suggestions from @eli-schwartz, in case other users need to do something similar:

... snip ...
if cuda_dep.found()
   # Get path to cuda lib64 dir (might be nicer way?)
    nvcc    = find_program('nvcc')
    cuda_lib64_path = nvcc.full_path().replace('/bin/nvcc', '/lib64')
    
    # Create pruned static library for CUFFT
    # TODO: recreate CUFFT_PTX_LATEST_ARCH from Makefile
    # (instead of hardcoding to known value as I've done here)
    arch_flags_pruned = cuda.nvcc_arch_flags('11', '8.0')
    arch_flags_pruned += cuda_arch_flags

    cufft_pruned = custom_target('libcufft_static_pruned',
        input: cuda_lib64_path / 'libcufft_static.a',
        output: 'libcufft_static_pruned.a',
        command: [nvprune, '@INPUT@', arch_flags_pruned, '-o', '@OUTPUT@'],
        install: true,
        install_dir:  meson.build_root()
    )

    # Now, compile cuFFT static library
    cufft_static = static_library('fft_kernels', 
               ['src/fft_kernels.cu', 'src/fft.cu'],
               cuda_args: ['-dc'], 
               include_directories: ['src'],
               override_options: ['buildtype=release'],
               build_by_default: true).extract_all_objects(recursive: false)
    
    # Next create _device_link.o
    device_link_flags = ['-dlink', 
                         '-Xcompiler', '"-march=native"',
                         '-Xcompiler', '-fPIC', 
                         '-Xcompiler', '-DPIC', 
                         '-L.', '-lcufft_static_pruned']
    device_link_flags += cuda_arch_flags

    cufft_device_link = custom_target('_device_link',
        input: cufft_static,
        output: '_device_link.o',
        command: [nvcc, '@INPUT@', device_link_flags, '-o', '@OUTPUT@'],
        install: true,
        install_dir:  meson.build_root()
    )
endif

# Compile main library
# Note cufft_device_link is added as a source (not as an objects:)
bifrost_lib = library('bifrost', [srcs_cpp, srcs_cu, cufft_device_link],              
                     include_directories: ['src'],
                     dependencies: [cuda_dep, found_deps],
                     install: true,
                     override_options: ['buildtype=release']
                    )

[eli-schwartz]

@obilaniu

Do you have any thoughts on integrating this handling into the official cuda module?