stumped trying to build the storage quickstart

[willie]

Background: I'm using cmake, which I have several projects building with, but I can't say I'm an expert. Also, this is my first time using a C++ manager like vcpkg. I'm building on macOS.

what works

I've been able to check out the repo, follow the instructions here to build, and run samples from that build.

what doesn't

When I try to build the quickstart, I get this error:

vcpkg % ./vcpkg install google-cloud-cpp[core,storage]
zsh: no matches found: google-cloud-cpp[core,storage]

If I ignore that error, and push onward, I get this error:

% cmake -H. -B.build -DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmake

CMake Error at /Users/willie/vcpkg/scripts/buildsystems/vcpkg.cmake:852 (_find_package):
  By not providing "Findgoogle_cloud_cpp_storage.cmake" in CMAKE_MODULE_PATH
  this project has asked CMake to find a package configuration file provided
  by "google_cloud_cpp_storage", but CMake did not find one.

  Could not find a package configuration file provided by
  "google_cloud_cpp_storage" with any of the following names:

    google_cloud_cpp_storageConfig.cmake
    google_cloud_cpp_storage-config.cmake

  Add the installation prefix of "google_cloud_cpp_storage" to
  CMAKE_PREFIX_PATH or set "google_cloud_cpp_storage_DIR" to a directory
  containing one of the above files.  If "google_cloud_cpp_storage" provides
  a separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  CMakeLists.txt:21 (find_package)

FWIW, my long term goal is to use the storage part of this library in a larger cmake project that already builds on macOS and Linux. (Before I used cmake, I tried to use the google-cloud-cpp library, but gave up and built a Go c-archive that I call from C++. It works, but now that I'm on cmake for the C++ build, I think it makes more sense to try and use the C++ library now.)

[devbww]

vcpkg % ./vcpkg install google-cloud-cpp[core,storage]
zsh: no matches found: google-cloud-cpp[core,storage]

The first thing you might want to try is defeating the matching failure by escaping the square brackets.

[willie]

I thought it was weird, but I was following the instructions at that link that state:

cd $HOME/vcpkg
./vcpkg install google-cloud-cpp[core,storage]

So I'm not really sure how to proceed from that.

[coryan]

Different shells behave differently. On most Unix shells that would just work. zsh is more strict (or less friendly). We should probably fix the README files.

[coryan]

If I ignore that error, and push onward, I get this error:

What @devbww said. That means the libraries did not get installed. I expect this will work:

FWIW, my long term goal is to use the storage part of this library in a larger cmake project that already builds on macOS and Linux.

Assuming you are not using vcpkg for the rest of your build, maybe you need to compile google-cloud-cpp from source. There are instructions for macOS at the the bottom of:

https://github.com/googleapis/google-cloud-cpp/blob/main/doc/packaging.md

I don't know if we have a test for it, but looks right to me.

[coryan]

Now, I'm just trying to remember enough of cmake to figure out find_package.

I think you are looking for: https://cmake.org/cmake/help/v3.26/variable/CMAKE_PREFIX_PATH.html

I've always used add_subdirectory for all of my compiled external dependencies.

Or maybe FetchContent?

https://cmake.org/cmake/help/latest/module/FetchContent.html

You may be able to do use it with google-cloud-cpp. We don't recommend it because it was really hard to test (do we test with all the deps installed the same way? Only some of them? All possible combinations?), but there is no reason to believe it would not work. You would need to satisfy all the dependencies google-cloud-cpp has.

(Maybe I'm overthinking this and should install them in the global directories like the defaults do, but I'm trying to keep it self contained. And another part of me saw how easy the bazel build was...)

If I may be so bold. I suggest you decide on one approach to manage your dependencies. All of the following can be made to work, with some tradeoffs:

• Use FetchContent and add_subdirectory(). Easy if you have a small number of deps, gets messier as you add more of them and they have recursive deps, used to (still may) have problems if two deps defined the same target name as target names are global.
• Use git submodules and add_subdirectory(). Some of the same tradeoffs.
  • Both of the previous approaches basically are trying to turn CMake into a dependency manager, and it is not that. And if I was going to write a dep manager, the CMake language is not the implementation language I would choose.
• Use a package manager (homebrew, vcpkg, Conda, Conan, apt, etc.) to install all your deps. Works great, but can take some time to learn a new tool (the dep manager). Can become a bottleneck if one of your deps is not in the dep manager and you have to create the port files / packaging instructions. Can be a problem if you chose a package manager tied to a single platform and you need to port.
• Manually installing the deps. Works, lots of manual fiddling with installation options.
• Try to mix and match two dep managers: don't. Things will break in terrifyingly complicated ways.
• Try to mix and match approaches: don't, that way lies madness.

[willie]

This is good advice. I'll followup on the cmake links. I admit I come across as a bit crazy and illogical with some of these questions, but I feel like it would useful to explain the build history of the project I'm trying to integrate this library into.

Pre 2007, in the beginning: handcrafted Xcode project on Mac for development, debug, and production on macOS on Xserves in our own datacenter. Some libraries were built with the Xcode project, some were built externally and linked statically. File access to content is over NFS.

Sometime, move to Linux: handcrafted Xcode project on Mac for development and debug, separate makefile for production Linux in our own datacenter. Some libraries were built with the Xcode project, some were built externally and linked statically. File access over NFS.

Circa 2018, we move to GYP, because it was too hard having separate xcode projects and makefiles: handcrafted GYP project files (gyp could generate xcode and makefiles), generated Xcode project on Mac for development and debug, generated Makefile for production Linux (in Docker) in GCP. Some libraries were built with the Xcode project and makefile, some were built externally and linked statically, although fewer on Linux as some were available on Linux via package management via Docker. File access over NFS.

2019, we move to GCS, I build a Go static library: handcrafted GYP project files, generated Xcode project on Mac for development and debug, generated Makefile for production Linux (in Docker) in GCP. Some libraries were built with the Xcode project and makefile, some were built externally and linked statically, although fewer on Linux as some were available on Linux via package management via Docker. _File access over GCS (content is too large to be hosted on NFS in GCP for the cost). This created a problem, as the Google Cloud C++ libraries were basically cmake only at the time and I didn't understand cmake then. I built my own GCS C static library with Go and linked against that. It worked good enough to use in production for 2 years.

2020, gyp is dead, move to cmake: handcrafted cmake project files (cmake finally has an xcode exporter that works), generated Xcode project on Mac for development and debug, generated Ninja for production Linux (in Docker) in GCP. Some libraries were converted over to cmake and then were built with the Xcode project and ninja, some were built externally and linked statically, although fewer on Linux as some were available on Linux via package management. File access over GCS is still via the Go static library I built.

In short, this project predates C++ package managers, lasted longer than a Google build system gyp, and continues to be essential today. I think things could work better if I moved to the C++ library since the rest of the project is C/C++ and there are occasional quirks of using a Go library from a C++ app in terms of debugging and memory. Given the context of all this, do you have different or more specific advice about what approach to take? My goal continues to be ease of debug and development, consistent deployment into production.

[coryan]

Given the context of all this, do you have different or more specific advice about what approach to take? My goal continues to be ease of debug and development, consistent deployment into production.

Take this with a grain of salt, because I am not that familiar with Xcode or the specific needs of your project.

I would aim to move the build towards a single package manager and CMake. You may not be able to get there in a single step, but it defines a direction and a way to make decisions about "how do I install X".

I will use vcpkg as an example, but I am sure analogous approaches can be found for other package managers.

In this approach your top-level project would have a manifest file declaring all your dependencies. When you configure the project CMake would invoke vcpkg to build (or most likely download from a binary cache) all your dependencies. In the CMake project files you would always write find_package(Foo REQUIRED CONFIG) to find your deps.

If a dependency does not have a vcpkg port you have a choice:

• Option 1: you write a portfile (well it is two files) that you keep in your own registry.
• Option 2: if that is too time consuming, maybe you temporarily write a small Foo-config.cmake file (so find_package(Foo ...) works). And use a pre-built binary. Over time you can work to reduce the number of such pre-built dependencies.

As I said, this may not work for you because I do not understand enough about Xcode and/or the details of your project.

[willie]

I appreciate the advice. It is tempting to use one of the package managers. For the time being, I'm going to try to use this library with CMake without the help of a package manager for the main project.

I wouldn't worry about Xcode in this particular setup, CMake is my single source of build truth and it exports Xcode projects just fine. I've never (knocks on wood) had something that would build with CMake that didn't also build with the exported Xcode project. I suppose package managers are a potential wrinkle here.

[willie]

Good news: I took your advice and tried out vcpkg to build this library into my project and it worked. It didn't take all that long for me to pull it in and use it. Of course, getting to a place where it "fit in" in a way that I liked took some time, but it's working great. Works with Xcode (via the cmake exporter), too. I still have some manually managed libraries, but they coexist just fine. So, thank you for nudging me that direction.

Bad news: Not everything I have been manually managing exists in vcpkg, so I'm living between worlds a bit. Also, docker builds on a fresh image are crazy slow. I haven't investigated the binary cache yet.