Building google-cloud-cpp//:storage using bazel fails on Mac because it cannot find the CA certificate

[sandeshbhatjr]

I want to use the google-cloud-cpp storage library on my Mac using Bazel. I have a simple C++ source code gcs.cc that depends on google/cloud/storage/client.h. I use the following Bazel build files to build this.

WORKSPACE

cc_binary(
    name = "gcs",
    srcs = ["gcs.cc"],
    deps = [
        "@com_github_googleapis_google_cloud_cpp//:storage",
    ],
)

BUILD (taken from quickstart)

# A minimal WORKSPACE file showing how to use the Google Cloud Storage C++
# client library in Bazel-based projects.
workspace(name = "uploader")

# Add the necessary Starlark functions to fetch google-cloud-cpp.
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_github_googleapis_google_cloud_cpp",
    sha256 = "e4e9eac1e7999eff195db270bc2a719004660b3730ebb5d2f444f2d2057e49b2",
    strip_prefix = "google-cloud-cpp-1.35.0",
    url = "https://github.com/googleapis/google-cloud-cpp/archive/v1.35.0.tar.gz",
)

# Load indirect dependencies due to
#     https://github.com/bazelbuild/bazel/issues/1943
load("@com_github_googleapis_google_cloud_cpp//bazel:google_cloud_cpp_deps.bzl", "google_cloud_cpp_deps")

google_cloud_cpp_deps()

load("@com_google_googleapis//:repository_rules.bzl", "switched_rules_by_language")

switched_rules_by_language(
    name = "com_google_googleapis_imports",
    cc = True,
    grpc = True,
)

load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")

grpc_deps()

load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")

grpc_extra_deps()

Running bazel build //:gcs --verbose_failures fails as follows:

INFO: Analyzed target //:gcs (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/bd96976c1318431bc51cfb4dcf0fa145/external/com_github_curl_curl/BUILD.bazel:60:8: Executing genrule @com_github_curl_curl//:gen-ca-bundle-linux failed: (Exit 1): bash failed: error executing command 
  (cd /private/var/tmp/_bazel_jogisararamachandrasandeshbhat/bd96976c1318431bc51cfb4dcf0fa145/sandbox/darwin-sandbox/216/execroot/uploader && \
  exec env - \
    PATH=/Users/jogisararamachandrasandeshbhat/Library/Caches/bazelisk/downloads/bazelbuild/bazel-5.0.0-darwin-arm64/bin:/Applications/google-cloud-sdk/bin:/Users/jogisararamachandrasandeshbhat/.nvm/versions/node/v16.13.1/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; 
      if [ -f /etc/fedora-release ]; then
        echo '\''#define CURL_CA_BUNDLE "/etc/pki/tls/certs/ca-bundle.crt"'\''
      elif [ -f /etc/redhat-release ]; then
        echo '\''#define CURL_CA_BUNDLE "/etc/pki/tls/certs/ca-bundle.crt"'\''
      elif [ -f /etc/debian_version ]; then
        echo '\''#define CURL_CA_BUNDLE "/etc/ssl/certs/ca-certificates.crt"'\''
      elif [ -f /etc/arch-release ]; then
        echo '\''#define CURL_CA_BUNDLE "/etc/ssl/certs/ca-certificates.crt"'\''
      else
        >&2 echo "Unknown platform, cannot guess location of CA bundle"
        exit 1
      fi >bazel-out/darwin_arm64-fastbuild/bin/external/com_github_curl_curl/include/curl_ca_bundle_location.h
    ')
# Configuration: cb0eb31a4999642048ec73e521abed0766b6a8a3d95f952f7b8fc8047cfa396e
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
Unknown platform, cannot guess location of CA bundle
Target //:gcs failed to build
INFO: Elapsed time: 0.761s, Critical Path: 0.44s
INFO: 64 processes: 11 internal, 53 darwin-sandbox.
FAILED: Build did NOT complete successfully

The issue seems to be that while build external dependency curl, the build file is not able to automatically figure out where the CA bundle is. It seems that Mac OS has an internal security mechanism and does not really have a CA bundle, but to remedy this- it is possible to install it by using port install curl-ca-bundle. Using port contents curl-ca-bundle, the address of the CA bundle can be found to be at:

Port curl-ca-bundle contains:
  /opt/local/etc/openssl/cert.pem
  /opt/local/share/curl/curl-ca-bundle.crt

My question is how to go from here to supply this path to the rule building curl in the process? Given the path above, how do I make the build work?

--
System details:
Hardware: MacBook Pro (14-inch, 2021), Apple M1 Pro, 32 GB RAM
OS: macOS Moneterey 12.1
Bazel: 5.0.0

[coryan]

Thank you for the detailed investigation and the system info.

The issue seems to be that while build external dependency curl, the build file is not able to automatically figure out where the CA bundle is. It seems that Mac OS has an internal security mechanism and does not really have a CA bundle,

On macOS we should not be trying to use the CA bundle, we have code to avoid this:

google-cloud-cpp/bazel/curl.BUILD

Lines 119 to 126 in 5660181

	cc_library(
	name = "define-ca-bundle-location",
	deps = select({
	":windows": [],
	":macos": [],
	"//conditions:default": [":define-ca-bundle-location-linux"],
	}),
	)

and in fact we do not use the curl_ca_bundle_location.h header on macOS:

google-cloud-cpp/bazel/curl.BUILD

Lines 491 to 501 in 5660181

	"#elif defined(__APPLE__)",
	"# define HAVE_FSETXATTR_6 1",
	"# define HAVE_SETMODE 1",
	"# define HAVE_SYS_FILIO_H 1",
	"# define HAVE_SYS_SOCKIO_H 1",
	"# define OS \"x86_64-apple-darwin15.5.0\"",
	"# define USE_SECTRANSP 1",
	"#else",
	"# if !defined(CURL_CA_BUNDLE)",
	"# include \"include/curl_ca_bundle_location.h\"",
	"# endif // CURL_CA_BUNDLE",

I think the bug is in how we detect if the platform is macOS:

google-cloud-cpp/bazel/curl.BUILD

Lines 22 to 26 in 5660181

	config_setting(
	name = "macos",
	values = {"cpu": "darwin"},
	visibility = ["//visibility:public"],
	)

You are using Bazel 5.0, I do not believe we have any tests for that, and I am guessing it the platform definition is wrong in that case. Let me see if I can confirm this.

[coryan]

Is not Bazel 5.0, I just tested that (using our google/cloud/storage/quickstart program). Maybe we have problems with the Apple chips. Let me summon @devjgm who understands the macOS ecosystem much better than I do.

[sandeshbhatjr]

Thanks for the quick reply. I wonder if the issue is that darwin_arm64 is somehow not included in the platform definition. I'm not really well-versed in platform dependent bazel builds, so I wait for @devjgm.

[devjgm]

Quick update: I can repro this on my mac mini w/ an M1 chip from within our repo building the //:storage target. Let me investigate further.

[devjgm]

I created #8078

[devjgm]

Thanks for reporting this issue. It should be fixed in #8101, which will be included in our release next week (Feb 2022).