Skip to content

Commit 3f7aa9e

Browse files
aiutoaiuto
committed
Start to bazel rules for curl
- mostly generated with configure2bazel - Fix IDN configure flags and update overlay - Add macos and linux arm overlays. - splice it in to the smoke test - generates working curl library of the right size for macos - not working for linux yet. There is a problem creating curl_static, where the cc toolchain is injectined a link arg that doesn't work. ``` /root/.cache/bazel/_bazel_root/81a15fa9a2846e82038a778136785275/external/gcc_toolchain++gcc_toolchains+gcc_toolchain_aarch64/bin/aarch64-unknown-linux-gnu-ld.bfd: unrecognized option '--start-lib' ``` There is some real ugly stuff going on to try to use the output of configure_make as shared libarries. - copying the ssl headers to a new place so we can make a cc_library This is needed because the `includes` attribute on cc_import doesn't work. We need to create a real cc_library to depend on to get that right. - the cc_library / cc_shared_library pair for everything Again, cc_import doesn't seem to work Look at /opt/datadog-agent on local laptop ``` -rwxr-xr-x 1 root wheel 4815040 Dec 9 14:57 /opt/datadog-agent/embedded/lib/libcrypto.3.dylib -rwxr-xr-x 1 root wheel 4857760 Dec 9 14:57 /opt/datadog-agent/embedded/lib/python3.13/site-packages/confluent_kafka/.dylibs/libcrypto.3.dylib -rwxr-xr-x 1 root wheel 4857760 Dec 9 14:57 /opt/datadog-agent/embedded/lib/python3.13/site-packages/psycopg_c/.dylibs/libcrypto.3.dylib -rwxr-xr-x 1 root wheel 965408 Dec 9 14:57 /opt/datadog-agent/embedded/lib/libssl.3.dylib -rwxr-xr-x 1 root wheel 946896 Dec 9 14:57 /opt/datadog-agent/embedded/lib/python3.13/site-packages/confluent_kafka/.dylibs/libssl.3.dylib -rwxr-xr-x 1 root wheel 946896 Dec 9 14:57 /opt/datadog-agent/embedded/lib/python3.13/site-packages/psycopg_c/.dylibs/libssl.3.dylib -rwxr-xr-x 1 root wheel 731904 Dec 9 14:57 /opt/datadog-agent/embedded/lib/python3.13/site-packages/confluent_kafka/.dylibs/libcurl.4.dylib ``` ``` ls -l bazel-bin/external/+_repo_rules+curl/libcurl.dylib bazel-bin/external/+_repo_rules+openssl/libimported_*dylib -r-xr-xr-x 1 tony.aiuto wheel 803968 Jan 13 23:02 bazel-bin/external/+_repo_rules+curl/libcurl.dylib -r-xr-xr-x 1 tony.aiuto wheel 4839472 Jan 13 16:54 bazel-bin/external/+_repo_rules+openssl/libimported_crypto_shared.dylib -r-xr-xr-x 1 tony.aiuto wheel 949008 Jan 13 22:06 bazel-bin/external/+_repo_rules+openssl/libimported_ssl_shared.dylib Note the similar sizes. Drift is probably because of nghttp2. Looking at the linking ``` $ otool -L /opt/datadog-agent/embedded/lib/python3.13/site-packages/confluent_kafka/.dylibs/libcurl.4.dylib /opt/datadog-agent/embedded/lib/python3.13/site-packages/confluent_kafka/.dylibs/libcurl.4.dylib: /DLC/confluent_kafka/.dylibs/libcurl.4.dylib (compatibility version 13.0.0, current version 13.0.0) @loader_path/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0) @loader_path/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0) @loader_path/libz.1.3.1.dylib (compatibility version 1.0.0, current version 1.3.1) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation ... /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices ... /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration ... /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2) $ otool -L bazel-bin/external/+_repo_rules+curl/libcurl.dylib bazel-bin/external/+_repo_rules+curl/libcurl.dylib: @rpath/libcurl.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/nghttp2.so.14 (compatibility version 0.0.0, current version 0.0.0) @rpath/libimported_crypto_shared.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libimported_ssl_shared.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libz.dylib (compatibility version 0.0.0, current version 0.0.0) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation ... /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration ... /usr/lib/libc++.1.dylib ... /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1356.0.0) ``` The extra CoreServices in the other version is unexpected, but may be a fault in the way Kafka builds it. fix deps
1 parent 6b75809 commit 3f7aa9e

File tree

15 files changed

+3110
-11
lines changed

15 files changed

+3110
-11
lines changed

‎.gitlab/bazel/build-deps.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,15 @@
77
bazel:build-deps:linux-amd64:
88
extends: [ .bazel:build-deps, .bazel:runner:linux-amd64 ]
99
script:
10+
- printenv
11+
- echo $PATH
1012
- bazel build //deps/... --platforms=//bazel/platforms:linux_x86_64
1113

1214
bazel:build-deps:linux-arm64:
1315
extends: [ .bazel:build-deps, .bazel:runner:linux-arm64 ]
1416
script:
17+
- printenv
18+
- echo $PATH
1519
- bazel build //deps/... --platforms=//bazel/platforms:linux_arm64
1620

1721
bazel:build-deps:macos-amd64:

‎BUILD.bazel‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,14 @@ config_setting(
3333
],
3434
)
3535

36+
platform(
37+
name = "linux_arm64_platform",
38+
constraint_values = [
39+
"@platforms//os:linux",
40+
"@platforms//cpu:arm64",
41+
],
42+
)
43+
3644
config_setting(
3745
name = "linux_arm64",
3846
constraint_values = [

‎MODULE.bazel‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,9 @@ include("//deps/compile_policy:compile_policy.MODULE.bazel")
200200
# buildifier: leave-alone
201201
include("//deps/cpython:cpython.MODULE.bazel")
202202

203+
# buildifier: leave-alone
204+
include("//deps/curl:curl.MODULE.bazel")
205+
203206
# buildifier: leave-alone
204207
include("//deps/gstatus:gstatus.MODULE.bazel")
205208

‎deps/BUILD.bazel‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ filegroup(
2828
"@platforms//os:windows": [
2929
"@cpython//:python_win",
3030
],
31+
"@platforms//os:macos": [
32+
"@curl",
33+
"@nghttp2",
34+
],
3135
"//conditions:default": [],
3236
}),
3337
visibility = [

‎deps/curl/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
curl-8.16.0
2+
curl-8.16.0.tar.gz

‎deps/curl/BUILD.bazel‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
load("@bazel_skylib//rules:diff_test.bzl", "diff_test")
2+
load("//bazel/tools:generate_module_bazel.bzl", "generate_module_bazel")
3+
4+
VERSION = "8.16.0"
5+
6+
generate_module_bazel(
7+
name = "curl_module_bazel_new",
8+
out = "curl.MODULE.bazel.new",
9+
module = "curl",
10+
sha256 = "a21e20476e39eca5a4fc5cfb00acf84bbc1f5d8443ec3853ad14c26b3c85b970",
11+
strip_prefix = "curl-%s" % VERSION,
12+
tags = ["manual"],
13+
target_compatible_with = ["@platforms//os:linux"],
14+
url = "https://curl.haxx.se/download/curl-%s.tar.gz" % VERSION,
15+
)
16+
17+
diff_test(
18+
name = "module_file_up_to_date_test",
19+
file1 = ":curl.MODULE.bazel.new",
20+
file2 = "curl.MODULE.bazel",
21+
target_compatible_with = ["@platforms//os:linux"],
22+
)

‎deps/curl/Makefile‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
C2B=$(HOME)/ws/experimental/teams/agent-supply-chain/configure2bazel
2+
PRISTINE=curl-8.16.0
3+
4+
next:
5+
echo Do both steps on each platform.
6+
echo Adjust overlay/overlay.BUILD.bazel
7+
echo Adjust overlay/lib/curl_config.h
8+
9+
macos1:
10+
python3 add_configuration.py \
11+
--pristine_dir=$(PRISTINE) \
12+
--configure_options=config_opts.txt
13+
echo Maybe: cd linux_aarch64 ; make
14+
echo Maybe: cd darwin_arm64 ; make
15+
16+
macos2:
17+
python3 $(C2B)/analyze.py \
18+
--pristine_dir=$(PRISTINE) \
19+
--configured_dir=darwin_arm64 \
20+
--configured_name=darwin_arm64 \
21+
--overlay=overlay
22+
git diff overlay/lib_contents.bzl
23+
24+
linux1:
25+
python3 add_configuration.py \
26+
--pristine_dir=$(PRISTINE) \
27+
--configured_name=linux_arm64 \
28+
--configure_options=config_opts.txt
29+
30+
linux2:
31+
python3 /tmp/analyze.py \
32+
--pristine_dir=$(PRISTINE) \
33+
--configured_dir=linux_arm64 \
34+
--configured_name=linux_arm64 \
35+
--overlay=overlay
36+
git diff overlay/lib_contents.bzl

‎deps/curl/config_opts.txt‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
--disable-manual
2+
--disable-debug
3+
--enable-optimize
4+
--disable-static
5+
--disable-ldap
6+
--disable-ldaps
7+
--disable-rtsp
8+
--enable-proxy
9+
--disable-dependency-tracking
10+
--enable-ipv6
11+
--without-applic-idn
12+
--without-libidn2
13+
--without-gnutls
14+
--without-librtmp
15+
--without-libssh2
16+
--without-libpsl
17+
--without-libuv
18+
--without-winidn
19+
--without-zsh-functions-dir
20+
--without-fish-functions-dir
21+
--with-ssl
22+
--with-zlib
23+
--with-nghttp2
24+
--disable-docs
25+
--disable-libcurl-option
26+
--disable-versioned-symbols

‎deps/curl/curl.MODULE.bazel‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# This file is generated. Do not hand edit.
2+
3+
http_archive = use_repo_rule("//third_party/bazel/tools/build_defs/repo:http.bzl", "http_archive")
4+
5+
http_archive(
6+
name = "curl",
7+
files = {
8+
"BUILD.bazel": "//deps/curl:overlay/overlay.BUILD.bazel",
9+
"darwin_arm64/lib/curl_config.h": "//deps/curl:overlay/darwin_arm64/lib/curl_config.h",
10+
"lib/curl_config.h": "//deps/curl:overlay/lib/curl_config.h",
11+
"lib_contents.bzl": "//deps/curl:overlay/lib_contents.bzl",
12+
"linux_arm64/lib/curl_config.h": "//deps/curl:overlay/linux_arm64/lib/curl_config.h",
13+
},
14+
sha256 = "e9274a5f8ab5271c0e0e6762d2fce194d5f98acc568e4ce816845b2dcc0cf88f",
15+
strip_prefix = "curl-8.18.0",
16+
url = "https://curl.haxx.se/download/curl-8.18.0.tar.gz",
17+
)

0 commit comments

Comments
 (0)