Skip to content

Commit f3b9e83

Browse files
robayuaiuto
robayu
and
aiuto
authored
[ABLD-340] Datadog Internal Artifact Registry Setup (#42969)
### What does this PR do? Adds Datadog Internal Artifact Registry config setup to have CI (which runs internally within Datadog Datacenter) pull from internal mirrors rather than going upstream (where possible, for python, go, java, npm) ### Motivation Remove reliance on upstream during dependency resolution where possible ### Describe how you validated your changes - [ ] CI Jobs pass - [ ] Test that this setup won't impact external contributors with no access to the Datadog internal artifact registries ### Additional Notes - The `.adms/bazel/adms.mirror.cfg` file blocks all URLs (except the whitelisted ones) in CI during `bazel build` commands, if these prove to be too blocking we will need to add more exceptions to the whitelist or remove these blocks and just leave the rewrites (to use the internal registries when pulling bazel dependencies) - The env vars set in the different `.adms/<ecosystem>/gitlab.yaml` files will make CI jobs use the internal artifact registries as well as set the same env vars in the `.adms/<ecosystem>/<ecosystem>.bazelrc` files when running in CI, but will leave these blank when running locally - External contributors should not be affected by this configuration and only Datadog CI builds should be affected - Internal Datadog employees can choose to set the same environment variables within an `.envrc` file if desired so their local package manager pulls and bazel commands also use the internal artifact registries Co-authored-by: aiuto <[email protected]>
1 parent c26cfbb commit f3b9e83

File tree

11 files changed

+198
-0
lines changed

11 files changed

+198
-0
lines changed

‎.adms/adms.bazelrc‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# DO NOT EDIT
2+
3+
# Pull in general ADMS-related configuration
4+
import %workspace%/.adms/bazel/bazel.bazelrc
5+
# Configure Python-related Bazel settings
6+
import %workspace%/.adms/python/python.bazelrc
7+
# Configure Go-related Bazel settings
8+
import %workspace%/.adms/go/go.bazelrc
9+
# Configure Npm-related Bazel settings
10+
import %workspace%/.adms/npm/npm.bazelrc

‎.adms/bazel/adms.mirror.cfg‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# DO NOT EDIT
2+
3+
# Datadog Internal Hosts
4+
#######################################################################
5+
# (Allow more such hosts as needed.)
6+
# Note that anything else we rewrite to must also be allowed here.
7+
allow binaries.ddbuild.io
8+
allow depot-read-api-bzl.us1.ddbuild.io
9+
allow depot-read-api-cli.us1.ddbuild.io
10+
allow depot-read-api-generic.us1.ddbuild.io
11+
allow depot-read-api-go.us1.ddbuild.io
12+
allow depot-read-api-java.us1.ddbuild.io
13+
allow depot-read-api-python.us1.ddbuild.io
14+
allow registry.ddbuild.io
15+
allow vault.us1.ddbuild.io
16+
allow nodejs.org
17+
allow go.dev
18+
allow go.dev/dl
19+
allow golang.google.cn
20+
allow tukaani.org
21+
allow bcr.bazel.build
22+
allow dd-agent-omnibus.s3.amazonaws.com
23+
allow lua.org
24+
allow sqlite.org
25+
allow download.savannah.nongnu.org
26+
allow ftp.rpm.org
27+
allow gnupg.org
28+
allow sourceware.org
29+
allow mirrors.kernel.org
30+
allow mirrors.edge.kernel.org
31+
32+
# DEBUGGING
33+
######################################################################
34+
# These hosts have been used for downloading Java artifacts. We'll allow them
35+
# for the time being, but we should dig into how they are being specified in
36+
# broader Bazel configuration, and work get these artifacts into ADMS more
37+
# formally, so we can begin to lock down this configuration more.
38+
39+
# Sqreen sources
40+
# (https://github.com/DataDog/logs-backend/blob/93eccaef54c4d0f99151c2bf49ff2bcb028c0bd7/rules/jvm/setup.bzl#L64-L73)
41+
allow sqreen-ci-java.s3.amazonaws.com
42+
43+
# dd-agent and dd-trace jars
44+
# (https://github.com/DataDog/logs-backend/blob/8ea7c29de6185481f61ba0b3affa97acad69d18b/rules/jvm/setup.bzl#L84-L98)
45+
allow s3.us-east-1.amazonaws.com
46+
######################################################################
47+
48+
# Rewrite Rules for External Hosts
49+
#######################################################################
50+
# The list of hosts below is based on what has already been observed in the wild,
51+
# and pulled in by Depot's Bazel support:
52+
#
53+
# aws-vault exec sso-build-stable-dependency-mgmt-s3-admin -- \
54+
# aws s3 ls s3://dd-depot-us1-ddbuild-io/bzl/
55+
#
56+
# We will explicitly enable downloading via Depot for these hosts, to preserve what
57+
# functionality we currently provide. This also gives us more control and visibility
58+
# into what we're actually downloading, and allows us to track down misconfigurations
59+
# more easily.
60+
rewrite (bcr.bazel.build)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
61+
rewrite (tukaani.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
62+
# rewrite (go.dev)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
63+
# rewrite (golang.google.cn)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
64+
rewrite (nodejs.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
65+
rewrite (aka.ms)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
66+
rewrite (amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
67+
rewrite (apache.jfrog.io)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
68+
rewrite (archive.apache.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
69+
rewrite (boostorg.jfrog.io)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
70+
rewrite (cache.agilebits.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
71+
rewrite (cdn.azul.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
72+
rewrite (fastdl.mongodb.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
73+
rewrite (get.helm.sh)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
74+
rewrite (git.kernel.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
75+
rewrite (github.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
76+
77+
# Taken from https://blog.aspect.build/configuring-bazels-downloader
78+
# > For some reason the bazel team decided that mirror.bazel.build should be the source of truth for these 3 files, let
79+
# > those through
80+
rewrite (mirror.bazel.build/bazel_coverage_output_generator/.*) depot-read-api-bzl.us1.ddbuild.io/$1
81+
rewrite (mirror.bazel.build/bazel_java_tools/.*) depot-read-api-bzl.us1.ddbuild.io/$1
82+
rewrite (mirror.bazel.build/openjdk/.*) depot-read-api-bzl.us1.ddbuild.io/$1
83+
# > For everything else, our urls exactly match what mirror.bazel.build gives so skip the indirection
84+
rewrite mirror.bazel.build/(.*) depot-read-api-bzl.us1.ddbuild.io/$1
85+
86+
rewrite (releases.hashicorp.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
87+
rewrite (snapshot.ubuntu.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
88+
rewrite (sourceware.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
89+
rewrite (storage.googleapis.com)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
90+
rewrite (www.colm.net)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
91+
rewrite (www.openssl.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
92+
rewrite (www.gnupg.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
93+
rewrite (www.unixodbc.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
94+
rewrite (mirrors.edge.kernel.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
95+
rewrite (pyyaml.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
96+
rewrite (mirrors.kernel.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
97+
98+
# Rewrite Rules for External Hosts - Specific Language Ecosystems
99+
#######################################################################
100+
# In some cases, we are accessing upstream repositories for various language ecosystems
101+
# that we otherwise support internally, despite Bazel being ostensibly configured to use
102+
# our internal repositories. This potentially suggests "holes" in our configuration, or
103+
# artifacts being pulled in via harder-to-detect means.
104+
#
105+
# These situations are highlighted here for visibility; eventually, we should sort out
106+
# these situations and entirely block the relevant upstream hosts entirely.
107+
108+
# TODO: Investigate why these aren't using our internal repository already.
109+
rewrite (pypi.org)/(.*) depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/$2
110+
rewrite (pypi.python.org)/(.*) depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/$2
111+
rewrite (files.pythonhosted.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
112+
113+
# TODO: Move to Depot NPM when available
114+
rewrite (registry.npmjs.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
115+
116+
# TODO: Move to Depot Java when we can determine why requests to repo1.maven.org
117+
# are still being made.
118+
rewrite (repo1.maven.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
119+
120+
# TODO: We don't support Rust in ADMS yet, but once we do, this can go away. In
121+
# the meantime, it's a kind of "back door" to mirroring Rust artifacts internally.
122+
rewrite (static.crates.io)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
123+
# This is for Rust runtimes
124+
rewrite (static.rust-lang.org)/(.*) depot-read-api-bzl.us1.ddbuild.io/$1/$2
125+
126+
# Block everything else
127+
#######################################################################
128+
# If something is not explicitly allowed above, it will be blocked. If users
129+
# need to download something that is blocked, they can speak with #dependency-management
130+
# in Slack to get it unblocked.
131+
block *
132+
all_blocked_message This blocking should only be happening in CI - reach out to Datadog#dependency-management

‎.adms/bazel/bazel.bazelrc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# DO NOT EDIT
2+
3+
# Ensure that Bazel uses the ADMS Bazel mirror for downloading dependencies.
4+
build:dd-internal --downloader_config=.adms/bazel/adms.mirror.cfg
5+
build:dd-internal --experimental_repository_downloader_retries=5

‎.adms/go/gitlab.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# DO NOT EDIT
2+
3+
variables:
4+
GOPROXY: "https://depot-read-api-go.us1.ddbuild.io/magicmirror/magicmirror/@current/|https://depot-read-api-go.us1.ddbuild.io/magicmirror/testing/@current/|https://depot-read-api-go.us1.ddbuild.io/magicmirror/magicmirror/@current/|https://depot-read-api-go.us1.ddbuild.io/magicmirror/testing/@current/|https://depot-read-api-go.us1.ddbuild.io/magicmirror/magicmirror/@current/|https://depot-read-api-go.us1.ddbuild.io/magicmirror/testing/@current/"
5+
GONOSUMDB: "github.com/DataDog,go.ddbuild.io"
6+
GOPRIVATE: ""

‎.adms/go/go.bazelrc‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# DO NOT EDIT
2+
3+
common --repo_env=GOPROXY
4+
common --repo_env=GONOSUMDB
5+
common --repo_env=GOPRIVATE

‎.adms/npm/gitlab.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# DO NOT EDIT
2+
3+
variables:
4+
YARN_NPM_REGISTRY_SERVER: "https://depot-read-api-npm.us1.ddbuild.io/internal/magicmirror/magicmirror"
5+
YARN_NPM_ALWAYS_AUTH: "false"
6+
NPM_CONFIG_REGISTRY: "https://depot-read-api-npm.us1.ddbuild.io/internal/magicmirror/magicmirror"
7+
npm_config_registry: "https://depot-read-api-npm.us1.ddbuild.io/internal/magicmirror/magicmirror"
8+
NPM_CONFIG_ALWAYS_AUTH: "false"
9+
npm_config_always_auth: "false"

‎.adms/npm/npm.bazelrc‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# DO NOT EDIT
2+
3+
# yarn Configuration
4+
common --repo_env=YARN_NPM_REGISTRY_SERVER
5+
common --repo_env=YARN_NPM_ALWAYS_AUTH
6+
7+
# npm Configuration
8+
common --repo_env=NPM_CONFIG_REGISTRY
9+
common --repo_env=npm_config_registry
10+
common --repo_env=NPM_CONFIG_ALWAYS_AUTH
11+
common --repo_env=npm_config_always_auth

‎.adms/python/gitlab.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# DO NOT EDIT
2+
3+
variables:
4+
PIP_INDEX_URL: "https://depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/simple"
5+
PIP_EXTRA_INDEX_URL: "https://depot-read-api-python.us1.ddbuild.io/magicmirror/testing/@current/simple"
6+
UV_INDEX: "https://depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/simple https://depot-read-api-python.us1.ddbuild.io/magicmirror/testing/@current/simple"
7+
UV_DEFAULT_INDEX: "https://depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/simple"

‎.adms/python/python.bazelrc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# DO NOT EDIT
2+
3+
common --repo_env=PIP_INDEX_URL
4+
common --repo_env=PIP_EXTRA_INDEX_URL
5+
common --repo_env=UV_INDEX
6+
common --repo_env=UV_DEFAULT_INDEX

‎.bazelrc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# Do not edit this file without a review from @DataDog/agent-build
22

3+
# Ensure access to DataDog internal artifact repositories in CI
4+
import %workspace%/.adms/adms.bazelrc
5+
36
# Startup options ------------------------------------------------------------------------------------------------------
47
startup --max_idle_secs=28800 # Keep the server alive for at most 8 hours of inactivity
58

@@ -53,5 +56,6 @@ build:ci --output_groups=+clippy_checks
5356
build:ci --aspects=@rules_rust//rust:defs.bzl%rustfmt_aspect
5457
build:ci --output_groups=+rustfmt_checks
5558

59+
#common:ci --config=dd-internal
5660
# Local development options --------------------------------------------------------------------------------------------
5761
try-import %workspace%/user.bazelrc

0 commit comments

Comments
 (0)