ci: Remove CI configurations for private repos and Fixes CI issues for macos (#112)

* fix e2e

Signed-off-by: mathetake <[email protected]>

* fix e2e

Signed-off-by: mathetake <[email protected]>

* Fix permission of install_docker.sh

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix to suite current convention of brew cask in install_docker.sh

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix brew cask command to the latest format

Signed-off-by: Kotaro Inoue <[email protected]>

* Revert "Fix brew cask command to the latest format"

This reverts commit bfccf6e.

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix a bit

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix a bit

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix to use local formulae downloaded from remote

Signed-off-by: Kotaro Inoue <[email protected]>

* Fix DEVELOPER.md

Signed-off-by: Kotaro Inoue <[email protected]>

Co-authored-by: mathetake <[email protected]>

Author: musaprg

.github/workflows/commit.yaml

@@ -54,15 +54,6 @@ jobs:
       run: make builders
 
     - name: "Run e2e tests using `getenvoy` and `e2e` binaries built by the upstream job"
-      env:
-        # Allow extension templates to have dependecies on private GitHub repositories.
-        E2E_ALLOW_PRIVATE_DEPENDENCIES: 'yes'
-        # Key of a GitHub "machine user" that has access to all private repositories needed by e2e tests:
-        #   user:           https://github.com/getenvoy-ci
-        #   key fingeprint: MD5:ae:8d:18:6a:1b:44:17:dc:b8:c8:0f:a9:48:44:be:2b
-        #
-        # See https://developer.github.com/v3/guides/managing-deploy-keys/#machine-users
-        E2E_GITHUB_MACHINE_USER_KEY: ${{ secrets.E2E_GITHUB_MACHINE_USER_KEY }}
       run: ./ci/e2e/linux/run_tests.sh
 
   e2e_macos:
@@ -91,13 +82,4 @@ jobs:
       run: make builders
 
     - name: "Run e2e tests using `getenvoy` and `e2e` binaries built by the upstream job"
-      env:
-        # Allow extension templates to have dependecies on private GitHub repositories.
-        E2E_ALLOW_PRIVATE_DEPENDENCIES: 'yes'
-        # Key of a GitHub "machine user" that has access to all private repositories needed by e2e tests:
-        #   user:           https://github.com/getenvoy-ci
-        #   key fingeprint: MD5:ae:8d:18:6a:1b:44:17:dc:b8:c8:0f:a9:48:44:be:2b
-        #
-        # See https://developer.github.com/v3/guides/managing-deploy-keys/#machine-users
-        E2E_GITHUB_MACHINE_USER_KEY: ${{ secrets.E2E_GITHUB_MACHINE_USER_KEY }}
       run: ./ci/e2e/macos/run_tests.sh

.github/workflows/release.yaml

@@ -123,9 +123,6 @@ jobs:
                                                                              # `getenvoy extension build | test | run` stable
 
     - name: "Run e2e tests using released `getenvoy` binary and published extension builder images"
-      env:
-        # Forbid extension templates to have dependecies on private GitHub repositories.
-        E2E_ALLOW_PRIVATE_DEPENDENCIES: 'no'
       run: ./ci/e2e/linux/run_tests.sh
 
   e2e_macos:
@@ -169,7 +166,4 @@ jobs:
                                                                              # `getenvoy extension build | test | run` stable
 
     - name: "Run e2e tests using released `getenvoy` binary and published extension builder images"
-      env:
-        # Forbid extension templates to have dependecies on private GitHub repositories.
-        E2E_ALLOW_PRIVATE_DEPENDENCIES: 'no'
       run: ./ci/e2e/macos/run_tests.sh

DEVELOPER.md

@@ -42,62 +42,9 @@ getenvoy extension init my-extension
 ```
 and follow the wizard.
 
-> NOTE: At the moment, `Rust` extensions have a dependency on a private `GitHub` repository [tetratelabs/envoy-wasm-rust-sdk](https://github.com/tetratelabs/envoy-wasm-rust-sdk).
->
-> In practice, it means that `Rust` toolchain (`cargo`) will have to pass through [GitHub authenticatation]() to be able to fetch the source code of [tetratelabs/envoy-wasm-rust-sdk](https://github.com/tetratelabs/envoy-wasm-rust-sdk).
->
-> For more details see a section on [SSH authentication](https://doc.rust-lang.org/cargo/appendix/git-authentication.html#ssh-authentication) in the [Cargo Book](https://doc.rust-lang.org/cargo/).
-
-To build a Wasm extension on `Mac OS`, do the following:
-1. [Configure SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent)
-2. Run:
-   ```shell
-   cd my-new-extension
-   
-   getenvoy extension build --toolchain-container-options \
-     '--mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock'
-   ```
-
 ### How to run e2e Tests
 
 Run:
 ```shell
 make e2e
 ```
-
-> NOTE: At the moment, `Rust` extensions have a dependency on a private `GitHub` repository [tetratelabs/envoy-wasm-rust-sdk](https://github.com/tetratelabs/envoy-wasm-rust-sdk).
->
-> In practice, it means that `Rust` toolchain (`cargo`) will have to pass through [GitHub authenticatation]() to be able to fetch the source code of [tetratelabs/envoy-wasm-rust-sdk](https://github.com/tetratelabs/envoy-wasm-rust-sdk).
->
-> For more details see a section on [SSH authentication](https://doc.rust-lang.org/cargo/appendix/git-authentication.html#ssh-authentication) in the [Cargo Book](https://doc.rust-lang.org/cargo/).
-
-To run e2e tests on `Mac OS`, do the following:
-1. [Configure SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent)
-2. Run:
-   ```shell
-   bash -c '
-   set -e
-
-   #
-   # restore original ownership over the SSH agent socket (mounted inside container)
-   #
-   trap "docker run --rm -t --mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock busybox chown 0 /run/host-services/ssh-auth.sock" EXIT
-
-   #
-   # pass ownership over the SSH agent socket (mounted inside container) to the current user
-   #
-   docker run --rm -t --mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock busybox chown $(id -u) /run/host-services/ssh-auth.sock
-
-   #
-   # Run e2e tests in the following context:
-   #  1. Pass SSH agent socket (so that build containers could download private dependencies)
-   #  2. Override location of Cargo cache (so that all extensions and their build containers could share the same cache)
-   #
-   E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS=" \
-   --mount type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock \
-   -e SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock \
-   -v /tmp/cache/getenvoy:/tmp/cache/getenvoy \
-   -e CARGO_HOME=/tmp/cache/getenvoy/extension/rust-builder/cargo" \
-   make e2e
-   '
-   ```

ci/e2e/linux/run_tests.sh

@@ -29,28 +29,6 @@ sudo chown -R $(id -u):$(id -g) "${E2E_CACHE_DIR}"
 # to speed up `getenvoy extension build|test`, re-use a single cache across all extensions created by e2e tests
 export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -v ${E2E_CACHE_DIR}:/tmp/cache/getenvoy -e CARGO_HOME=/tmp/cache/getenvoy/extension/rust-builder/cargo"
 
-forward_ssh_agent=false
-case "${E2E_ALLOW_PRIVATE_DEPENDENCIES}" in
-	yes | on | true | 1) forward_ssh_agent=true ;;
-esac
-
-if [[ "${forward_ssh_agent}" == "true" ]]; then
-	# setup SSH key that will be used by build containers to fetch private dependencies
-	mkdir -p $HOME/.ssh/
-	echo "${E2E_GITHUB_MACHINE_USER_KEY}" | base64 -d > $HOME/.ssh/id_rsa_e2e_github_machine_user
-	chmod 600 $HOME/.ssh/id_rsa_e2e_github_machine_user
-
-	# use a dedicated SSH agent to manage the keys needed by extension build containers
-	eval $(ssh-agent -s)
-	# always kill that SSH agent in the end
-	trap "ssh-agent -k" EXIT
-	# load a single key of a GitHub "machine user" that has access to all private repositories needed by e2e tests
-	ssh-add $HOME/.ssh/id_rsa_e2e_github_machine_user
-
-	# forward SSH agent into extension build containers so that to they could fetch source code from private GitHub repositories
-	export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -v ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK} -e SSH_AUTH_SOCK"
-fi
-
 # restore executable bit that get lost by Github Actions during artifact upload/download
 chmod a+x ${WORKSPACE_DIR}/build/bin/linux/amd64/*
 

ci/e2e/macos/install_docker.sh

@@ -16,11 +16,16 @@
 
 set -e
 
+TMP_DIR=$(mktemp -d)
+
 # Docker for Mac 2.0.0.3-ce-mac81,31259 (the last version of 'Docker for Mac' that can be installed in CI environment)
 E2E_MACOS_DOCKER_CASK_VERSION="${E2E_MACOS_DOCKER_CASK_VERSION:-8ce4e89d10716666743b28c5a46cd54af59a9cc2}"
 
 # install Docker for Mac
-brew cask install https://raw.githubusercontent.com/Homebrew/homebrew-cask/${E2E_MACOS_DOCKER_CASK_VERSION}/Casks/docker.rb
+pushd "${TMP_DIR}"
+curl -L https://raw.githubusercontent.com/Homebrew/homebrew-cask/${E2E_MACOS_DOCKER_CASK_VERSION}/Casks/docker.rb > docker.rb
+brew install --cask docker.rb
+popd
 
 # follow instructions from:
 #   https://github.com/microsoft/azure-pipelines-image-generation/issues/738#issuecomment-496211237

ci/e2e/macos/run_tests.sh

@@ -24,53 +24,13 @@ E2E_CACHE_DIR="${E2E_CACHE_DIR:-$HOME/cache/getenvoy}"
 # make sure the cache directory is first created on behalf of the current user
 mkdir -p "${E2E_CACHE_DIR}"
 
+# TODO: support multiple language
 # to speed up `getenvoy extension build|test`, re-use a single cache across all extensions created by e2e tests
 export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -v ${E2E_CACHE_DIR}:/tmp/cache/getenvoy -e CARGO_HOME=/tmp/cache/getenvoy/extension/rust-builder/cargo"
 
 # set HOME directory
 export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -e HOME=/tmp/getenvoy"
 
-forward_ssh_agent=false
-case "${E2E_ALLOW_PRIVATE_DEPENDENCIES}" in
-	yes | on | true | 1) forward_ssh_agent=true ;;
-esac
-
-if [[ "${forward_ssh_agent}" == "true" ]]; then
-	# unfortunately, older versions of 'Docker for Mac' (that can be installed in CI environment)
-	# do not support SSH agent forwarding.
-	# that is why we have to take care of it manually and work around the limitation that it's not
-	# possible to mount a Unix socket from a Mac host into a container
-
-	# setup SSH key that will be used by build containers to fetch private dependencies
-	mkdir -p $HOME/.ssh/
-	echo "${E2E_GITHUB_MACHINE_USER_KEY}" | base64 -D > $HOME/.ssh/id_rsa_e2e_github_machine_user
-	chmod 600 $HOME/.ssh/id_rsa_e2e_github_machine_user
-
-	# create a wrapper script around the original container entrypoint to setup SSH agent inside the container
-	echo '#!/usr/bin/env bash
-set -e
-# use an SSH agent to manage the keys (works better than a plain SSH key in case of Cargo)
-eval $(ssh-agent -s)
-# always kill that SSH agent in the end
-trap "ssh-agent -k" EXIT
-# load a single key of a GitHub "machine user" that has access to all private repositories needed by e2e tests
-ssh-add $HOME/.ssh/id_rsa
-# sanity check
-ssh-add -l
-
-# call the original entrypoint
-/usr/local/getenvoy/extension/builder/entrypoint.sh "$@"
-' > /tmp/entrypoint-wrapper.sh
-	chmod a+x /tmp/entrypoint-wrapper.sh
-
-	# mount SSH key into extension build containers
-	export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -v $HOME/.ssh/id_rsa_e2e_github_machine_user:/tmp/getenvoy/.ssh/id_rsa"
-	# mount the entrypoint wrapper script
-	export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} -v /tmp/entrypoint-wrapper.sh:/tmp/getenvoy/entrypoint-wrapper.sh"
-	# substitute entrypoint with a wrapper script
-	export E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS="${E2E_BUILTIN_TOOLCHAIN_CONTAINER_OPTIONS} --entrypoint /tmp/getenvoy/entrypoint-wrapper.sh"
-fi
-
 # restore executable bit that get lost by Github Actions during artifact upload/download
 chmod a+x ${WORKSPACE_DIR}/build/bin/darwin/amd64/*