[C++] Findzstd.cmake doesnt find zstd on Ubuntu 20.04

[asfimport]

I'm unable to use the pre-built c++ arrow libraries with a project using a CMakeList.txt containing

find_package(Arrow REQUIRED)

Giving the following error

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find zstd (missing: ZSTD_LIB)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/lib/x86_64-linux-gnu/cmake/arrow/Findzstd.cmake:82 (find_package_handle_standard_args)
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/lib/x86_64-linux-gnu/cmake/arrow/ArrowConfig.cmake:96 (find_dependency)
CMakeLists.txt:12 (find_package)

libzstd and libstd-dev are both installed

 

Environment: Ubuntu 20.04.2
Reporter: Nick Hortovanyi

_{Note: This issue was originally created as ARROW-13334. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Could you show ls -lah /usr/lib/x86_64-linux-gnu/?

[asfimport]

Nick Hortovanyi:

❯ ls -lah /usr/lib/x86_64-linux-gnu/|grep -i zstd
 -rw-r--r-- 1 root root 853K Mar 4 01:47 libzstd.a
 lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so -> libzstd.so.1.4.4
 lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so.1 -> libzstd.so.1.4.4
 -rw-r--r-- 1 root root 671K Mar 4 01:47 libzstd.so.1.4.4

[asfimport]

Nick Hortovanyi:
Spent hours trying to solve this. Even tried from source. Same issue. I tried to not use zstd at all.

Installed Facebook's Folly

and have even tried to include this a submodule of my project

https://github.com/facebook/folly/blob/master/CMake/FindZstd.cmake

 

Presently worked out how to set ZSTD_LIB with ZSTD_LIBRARY

LIST(INSERT CMAKE_MODULE_PATH 0 ${CMAKE_CURRENT_SOURCE_DIR}/CMake)
find_package(Zstd REQUIRED)
if (ZSTD_FOUND)
  set(ZSTD_LIB ${ZSTD_LIBRARY})
endif (ZSTD_FOUND
find_package(Arrow REQUIRED)

which works ...
However now it cant find re2 and ares per the following
...

CMake Warning at /usr/local/lib/cmake/arrow/Findre2Alt.cmake:25 (find_package):
By not providing "Findre2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "re2", but
CMake did not find one.
 
Could not find a package configuration file provided by "re2" with any of
the following names:
 
re2Config.cmake
re2-config.cmake
 
Add the installation prefix of "re2" to CMAKE_PREFIX_PATH or set "re2_DIR"
to a directory containing one of the above files. If "re2" provides a
separate development package or SDK, be sure it has been installed.

Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
 
 
CMake Warning at /usr/local/lib/cmake/arrow/Findc-aresAlt.cmake:25 (find_package):
By not providing "Findc-ares.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "c-ares", but
CMake did not find one.
 
Could not find a package configuration file provided by "c-ares" with any
of the following names:
 
c-aresConfig.cmake
c-ares-config.cmake
 
Add the installation prefix of "c-ares" to CMAKE_PREFIX_PATH or set
"c-ares_DIR" to a directory containing one of the above files. If "c-ares"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)

[asfimport]

Kouhei Sutou / @kou:
Does https://github.com/apache/arrow/tree/master/cpp/examples/minimal_build work with the official Ubuntu 20.04 libarrow-dev package?
https://arrow.apache.org/install/

[asfimport]

Nick Hortovanyi:
You can try it with docker?? 

 

I've had this working previously. But havent touched the Apache Arrow C++ code for 6 months+ 

[asfimport]

Kouhei Sutou / @kou:
I did. It works without any problem.

[asfimport]

Nick Hortovanyi:
Only built this new server a couple of weeks ago

The cmake files that arrow is looking for, simply don’t exist.

They aren’t installed by the package maintainers as they do not provide them

Therefore I can’t build a cmake project against Apache Arrow for c++ on Ubuntu 20.04.2

I can install the libraries from the repo and/or via source. Just can’t build

[asfimport]

Kouhei Sutou / @kou:
Could you create a Dockerfile that reproduces this case?

[asfimport]

Nick Hortovanyi:
Just look at the packages I’ve mentioned. They don’t install CMake files

It’s self evident

[asfimport]

Kouhei Sutou / @kou:
What are the packages you've mentioned? libarrow-dev?
https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/focal/main/a/apache-arrow/libarrow-dev_4.0.1-1_amd64.deb

libzstd-dev?

[asfimport]

Nick Hortovanyi:
I uninstalled those packages and installed from source https://arrow.apache.org/docs/developers/cpp/building.html

[asfimport]

Nick Hortovanyi:
dpkg -L libzstd-dev
/.
/usr
/usr/include
/usr/include/zbuff.h
/usr/include/zdict.h
/usr/include/zstd.h
/usr/include/zstd_errors.h
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libzstd.a
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/libzstd.pc
/usr/share
/usr/share/doc
/usr/share/doc/libzstd-dev
/usr/share/doc/libzstd-dev/README.Debian
/usr/share/doc/libzstd-dev/copyright
/usr/share/doc/libzstd-dev/examples
/usr/share/doc/libzstd-dev/examples/Makefile
/usr/share/doc/libzstd-dev/examples/README.md
/usr/share/doc/libzstd-dev/examples/common.h
/usr/share/doc/libzstd-dev/examples/dictionary_compression.c
/usr/share/doc/libzstd-dev/examples/dictionary_decompression.c
/usr/share/doc/libzstd-dev/examples/multiple_simple_compression.c
/usr/share/doc/libzstd-dev/examples/multiple_streaming_compression.c
/usr/share/doc/libzstd-dev/examples/simple_compression.c
/usr/share/doc/libzstd-dev/examples/simple_decompression.c
/usr/share/doc/libzstd-dev/examples/streaming_compression.c
/usr/share/doc/libzstd-dev/examples/streaming_decompression.c
/usr/share/doc/libzstd-dev/examples/streaming_memory_usage.c
/usr/lib/x86_64-linux-gnu/libzstd.so
/usr/share/doc/libzstd-dev/changelog.Debian.gz

[asfimport]

Nick Hortovanyi:
dpkg -s libzstd-dev
Package: libzstd-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 1079
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: amd64
Multi-Arch: same
Source: libzstd
Version: 1.4.4+dfsg-3ubuntu0.1
Depends: libzstd1 (= 1.4.4+dfsg-3ubuntu0.1)
Description: fast lossless compression algorithm – development files
Zstd, short for Zstandard, is a fast lossless compression algorithm, targeting
real-time compression scenarios at zlib-level compression ratio.
.
This package contains the headers and static library.
Homepage: https://github.com/facebook/zstd
Original-Maintainer: Debian Med Packaging Team debian-med-packaging@lists.alioth.debian.org

[asfimport]

Kouhei Sutou / @kou:
Our Findzstd.cmake doesn't require CMake files for Zstandard: https://github.com/apache/arrow/blob/master/cpp/cmake_modules/Findzstd.cmake#L73-L75
So libzstd-dev isn't a problem.

Installing pkg-config may help you.

I can't help more unless you provide reproducible information such as Dockerfile. Sorry.

[asfimport]

Nick Hortovanyi:
The Apache Arrow cmake files dont work with .pc files

pkg-config is installed

 
❯ dpkg -s pkg-config
Package: pkg-config
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 190
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Multi-Arch: foreign
Version: 0.29.1-0ubuntu4
Replaces: pkg-config-bin
Depends: libc6 (>= 2.14), libglib2.0-0 (>= 2.16.0), libdpkg-perl
Suggests: dpkg-dev
Conflicts: pkg-config-bin
Conffiles:
/etc/dpkg/dpkg.cfg.d/pkg-config-hook-config 467c877de00898418a9b26d432a1719a
Description: manage compile and link flags for libraries
pkg-config is a system for managing library compile and link flags that
works with automake and autoconf.
.
Increasingly libraries ship with ".pc" files that allow querying of the
compiler and linker flags needed to use them through the pkg-config(1)
program.
Original-Maintainer: Tollef Fog Heen <tfheen@debian.org>
Homepage: http://pkg-config.freedesktop.org