V2 upgrades without manager (#1757)

* V2 upgrades without manager

* overrides for ha migration

* fix build

* clean error message when upgrading addons and extensions

* include logs in support bundle

* record upgrade failures

* run all v1 install tests in v2 (#1775)

* remove TestInstallWithoutEmbed

* use new install

* check postupgrade override labels

* fix extension upgrade messages

* further improve wording

* fix nil map issue when upgrading operator

* f

* Update pkg/extensions/upgrade.go

* Update pkg/extensions/upgrade.go

---------

Co-authored-by: Salah Al Saleh <[email protected]>

* yaml v3

* handle panics

* use slog

* replace join with join2 (#1780)

* fix build

* fix ha join

* fix lint (#1781)

* add comments to ignore deprecated BuiltinConfigs field

* fix linting

* mount charts dir

* feat(v2): enable v2 restore (#1776)

* fix upgrades

* fix TestLocalArtifactMirror and TestCommandsRequireSudo (#1782)

* chore: no spinner if no terminal (#1778)

* fix panic in TestHostPreflightCustomSpec

* fix compile

* require license for install2

* f

* f

* use single-node-install.sh in LAM e2e test

* improve sudo test logging

* ???

* see if it's just one thing failing in sudo test

* use join2 command in 'node join'. Hide 'node' commands

* improve how we are requiring the license flag for installation

---------

Co-authored-by: Ethan Mosbaugh <[email protected]>

* chore(ci): fix TestGetInstallation migrate data dirs (#1786)

* feat(v2): fix e2e TestVersion (#1784)

* feat(v2): fix e2e TestVersion

* f

* f

* fix operator upgrade in airgap

* fix velero airgap upgrades

* fix airgap upgrades

* chore(v2): e2e test multiple upgrades (#1783)

* chore(v2): e2e test multiple upgrades

* f

* f

* f

* f

* f

* f

* f

* remove use of ecUpgrade2TargetVersion

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

* f

---------

Co-authored-by: Andrew Lavery <[email protected]>

* fix TestHostPreflightCustomSpec and TestUnsupportedOverrides (#1787)

* create an app release to test unsupported overrides with

* f

* simplify unsupported overrides test

* update TestHostPreflightCustomSpec to use real releases

* f

* disable old installations

* better logging

* remove operator/pkg/charts as it is now unused (#1795)

* remove operator/pkg/charts as it is now unused

* run the operator even when the version does not match

* add unit tests

* feat(v2): support for restore (#1794)

* feat(v2): support for restore

* feat(v2): support for restore

* feat(v2): support for restore

* f

* f

* f

* check chart contents directly instead of using k0s chart object (#1798)

* check chart contents directly instead of using k0s chart object

* simplify by no longer checking chart values, just results

* feat(v2): restore ha (#1797)

* feat(v2): restore ha

* f

* f

* f

* Fix dryrun tests (#1800)

* disable TestOldVersionUpgrade test

* feat(v2): fix extensions apply (#1799)

* fix validate-success

---------

Co-authored-by: Andrew Lavery <[email protected]>
Co-authored-by: Ethan Mosbaugh <[email protected]>

Author: 3 people

.github/workflows/ci.yaml

@@ -64,7 +64,6 @@ jobs:
     runs-on: ubuntu-latest
     needs:
       - int-tests-kind
-      - int-tests-manager
     steps:
       - name: Succeed if all tests passed
         run: echo "Integration tests succeeded"
@@ -89,21 +88,6 @@ jobs:
         run: |
           make -C tests/integration test-kind
 
-  int-tests-manager:
-    name: Integration tests (manager)
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Setup go
-        uses: actions/setup-go@v5
-        with:
-          go-version-file: go.mod
-          cache-dependency-path: "**/*.sum"
-      - name: Run tests
-        run: |
-          make -C tests/integration test-manager
-
   dryrun-tests:
     name: Dryrun tests
     runs-on: ubuntu-latest
@@ -587,6 +571,24 @@ jobs:
           export APP_VERSION="appver-${SHORT_SHA}"
           export RELEASE_YAML_DIR=e2e/kots-release-install
           ./scripts/ci-release-app.sh
+          
+          # then install a version with alternate unsupported overrides
+          export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')"
+          export APP_VERSION="appver-${SHORT_SHA}-unsupported-overrides"
+          export RELEASE_YAML_DIR=e2e/kots-release-unsupported-overrides
+          ./scripts/ci-release-app.sh
+          
+          # then install a version with additional failing host preflights
+          export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')"
+          export APP_VERSION="appver-${SHORT_SHA}-failing-preflights"
+          export RELEASE_YAML_DIR=e2e/kots-release-install-failing-preflights
+          ./scripts/ci-release-app.sh
+          
+          # then install a version with additional warning host preflights
+          export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')"
+          export APP_VERSION="appver-${SHORT_SHA}-warning-preflights"
+          export RELEASE_YAML_DIR=e2e/kots-release-install-warning-preflights
+          ./scripts/ci-release-app.sh
 
           # promote a release with improved dr support
           export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')-legacydr"
@@ -640,6 +642,12 @@ jobs:
           export RELEASE_YAML_DIR=e2e/kots-release-install-legacydr
           ./scripts/ci-release-app.sh
 
+          # then a noop upgrade
+          export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')"
+          export APP_VERSION="appver-${SHORT_SHA}-noop"
+          export RELEASE_YAML_DIR=e2e/kots-release-install
+          ./scripts/ci-release-app.sh
+
           # and finally an app upgrade
           export EC_VERSION="$(git describe --tags --abbrev=4 --match='[0-9]*.[0-9]*.[0-9]*')-upgrade"
           export APP_VERSION="appver-${SHORT_SHA}-upgrade"
@@ -701,10 +709,9 @@ jobs:
           - TestSingleNodeUpgradePreviousStable
           - TestInstallFromReplicatedApp
           - TestUpgradeFromReplicatedApp
-          - TestInstallWithoutEmbed
           - TestUpgradeEC18FromReplicatedApp
           - TestResetAndReinstall
-          - TestOldVersionUpgrade
+          # - TestOldVersionUpgrade
           - TestInstallSnapshotFromReplicatedApp
           - TestMultiNodeInstallation
           - TestMultiNodeHAInstallation
@@ -885,243 +892,6 @@ jobs:
           k0s-version-previous-stable: ${{ needs.find-previous-stable.outputs.k0s_version }}
           version-specifier: ${{ needs.export-version-specifier.outputs.version_specifier }}
 
-
-  # e2e-docker-install2 runs the e2e tests inside a docker container rather than a full VM
-  e2e-docker-install2:
-    name: E2E Docker (install2)
-    runs-on: ubuntu-latest
-    needs:
-      - git-sha
-      - build-current
-      - build-legacydr
-      - build-previous-k0s
-      - build-upgrade
-      - find-previous-stable
-      - release-app
-      - export-version-specifier
-    strategy:
-      fail-fast: false
-      matrix:
-        test:
-          - TestSingleNodeInstall2UpgradeUbuntuJammy
-          - TestSingleNodeInstall2UpgradeAlmaLinux8
-          - TestSingleNodeInstall2UpgradeDebian11
-          - TestSingleNodeInstall2UpgradeDebian12
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Download binary
-        uses: actions/download-artifact@v4
-        with:
-          name: current-release
-          path: output/bin
-      - name: Download new DR binary
-        uses: actions/download-artifact@v4
-        with:
-          name: legacydr-release
-          path: output/bin
-      - name: Setup go
-        uses: actions/setup-go@v5
-        with:
-          go-version-file: go.mod
-          cache-dependency-path: "**/*.sum"
-      - name: Login to DockerHub to avoid rate limiting
-        uses: docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKERHUB_USER }}
-          password: ${{ secrets.DOCKERHUB_PASSWORD }}
-      - name: Free up runner disk space
-        uses: ./.github/actions/free-disk-space
-      - name: Write license files
-        run: |
-          echo "${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE }}" | base64 --decode > e2e/license.yaml
-          echo "${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}" | base64 --decode > e2e/snapshot-license.yaml
-      - name: Run test
-        env:
-          SHORT_SHA: dev-${{ needs.git-sha.outputs.git_sha }}
-          LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE_ID }}
-          AIRGAP_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
-          SNAPSHOT_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
-          AIRGAP_SNAPSHOT_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_SNAPSHOT_LICENSE_ID }}
-          DR_AWS_S3_ENDPOINT: https://s3.amazonaws.com
-          DR_AWS_S3_REGION: us-east-1
-          DR_AWS_S3_BUCKET: kots-testim-snapshots
-          DR_AWS_S3_PREFIX: ${{ matrix.test }}-${{ github.run_id }}-${{ github.run_attempt }}
-          DR_AWS_S3_PREFIX_AIRGAP: ${{ matrix.test }}-${{ github.run_id }}-${{ github.run_attempt }}-airgap
-          DR_AWS_ACCESS_KEY_ID: ${{ secrets.TESTIM_AWS_ACCESS_KEY_ID }}
-          DR_AWS_SECRET_ACCESS_KEY: ${{ secrets.TESTIM_AWS_SECRET_ACCESS_KEY }}
-          EXPECT_K0S_VERSION: ${{ needs.build-current.outputs.k0s_version }}
-          EXPECT_K0S_VERSION_PREVIOUS: ${{ needs.build-previous-k0s.outputs.k0s_version }}
-          EXPECT_K0S_VERSION_PREVIOUS_STABLE: ${{ needs.find-previous-stable.outputs.k0s_version }}
-        run: |
-          make e2e-test TEST_NAME=${{ matrix.test }}
-      - name: Troubleshoot
-        if: ${{ !cancelled() }}
-        uses: ./.github/actions/e2e-troubleshoot
-        with:
-          test-name: '${{ matrix.test }}'
-
-  # e2e-docker-install2-failing runs the e2e tests inside a docker container rather than a full VM
-  # the tests here are expected to fail and should be moved to e2e-docker-install2 once they are fixed
-  e2e-docker-install2-failing:
-    name: E2E Docker (install2, failing)
-    if: contains(github.event.pull_request.labels.*.name, 'run-failing-install2-tests')
-    runs-on: ubuntu-latest
-    needs:
-      - git-sha
-      - build-current
-      - build-legacydr
-      - build-previous-k0s
-      - build-upgrade
-      - find-previous-stable
-      - release-app
-      - export-version-specifier
-    strategy:
-      fail-fast: false
-      matrix:
-        test:
-        - TestAlwaysFail
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Download binary
-        uses: actions/download-artifact@v4
-        with:
-          name: current-release
-          path: output/bin
-      - name: Download new DR binary
-        uses: actions/download-artifact@v4
-        with:
-          name: legacydr-release
-          path: output/bin
-      - name: Setup go
-        uses: actions/setup-go@v5
-        with:
-          go-version-file: go.mod
-          cache-dependency-path: "**/*.sum"
-      - name: Login to DockerHub to avoid rate limiting
-        uses: docker/login-action@v3
-        with:
-          username: ${{ secrets.DOCKERHUB_USER }}
-          password: ${{ secrets.DOCKERHUB_PASSWORD }}
-      - name: Free up runner disk space
-        uses: ./.github/actions/free-disk-space
-      - name: Write license files
-        run: |
-          echo "${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE }}" | base64 --decode > e2e/license.yaml
-          echo "${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}" | base64 --decode > e2e/snapshot-license.yaml
-      - name: Run test
-        env:
-          SHORT_SHA: dev-${{ needs.git-sha.outputs.git_sha }}
-          LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE_ID }}
-          AIRGAP_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
-          SNAPSHOT_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
-          AIRGAP_SNAPSHOT_LICENSE_ID: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_SNAPSHOT_LICENSE_ID }}
-          DR_AWS_S3_ENDPOINT: https://s3.amazonaws.com
-          DR_AWS_S3_REGION: us-east-1
-          DR_AWS_S3_BUCKET: kots-testim-snapshots
-          DR_AWS_S3_PREFIX: ${{ matrix.test }}-${{ github.run_id }}-${{ github.run_attempt }}
-          DR_AWS_S3_PREFIX_AIRGAP: ${{ matrix.test }}-${{ github.run_id }}-${{ github.run_attempt }}-airgap
-          DR_AWS_ACCESS_KEY_ID: ${{ secrets.TESTIM_AWS_ACCESS_KEY_ID }}
-          DR_AWS_SECRET_ACCESS_KEY: ${{ secrets.TESTIM_AWS_SECRET_ACCESS_KEY }}
-          EXPECT_K0S_VERSION: ${{ needs.build-current.outputs.k0s_version }}
-          EXPECT_K0S_VERSION_PREVIOUS: ${{ needs.build-previous-k0s.outputs.k0s_version }}
-          EXPECT_K0S_VERSION_PREVIOUS_STABLE: ${{ needs.find-previous-stable.outputs.k0s_version }}
-        run: |
-          make e2e-test TEST_NAME=${{ matrix.test }}
-      - name: Troubleshoot
-        if: ${{ !cancelled() }}
-        uses: ./.github/actions/e2e-troubleshoot
-        with:
-          test-name: '${{ matrix.test }}'
-
-  e2e-install2:
-    name: E2E Install2
-    runs-on: ${{ matrix.runner || 'ubuntu-22.04' }}
-    needs:
-      - build-current
-      - build-legacydr
-      - build-previous-k0s
-      - build-upgrade
-      - find-previous-stable
-      - release-app
-      - export-version-specifier
-    strategy:
-      fail-fast: false
-      matrix:
-        test:
-          - TestSingleNodeAirgapInstall2
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Download current binary
-        uses: actions/download-artifact@v4
-        with:
-          name: current-release
-          path: output/bin
-
-      - uses: ./.github/actions/e2e
-        with:
-          test-name: '${{ matrix.test }}'
-          is-large-runner: ${{ matrix.runner == 'embedded-cluster' }}
-          airgap-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
-          snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
-          snapshot-license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}
-          airgap-snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_SNAPSHOT_LICENSE_ID }}
-          license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE_ID }}
-          license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE }}
-          dr-aws-access-key-id: ${{ secrets.TESTIM_AWS_ACCESS_KEY_ID }}
-          dr-aws-secret-access-key: ${{ secrets.TESTIM_AWS_SECRET_ACCESS_KEY }}
-          k0s-version: ${{ needs.build-current.outputs.k0s_version }}
-          k0s-version-previous: ${{ needs.build-previous-k0s.outputs.k0s_version }}
-          k0s-version-previous-stable: ${{ needs.find-previous-stable.outputs.k0s_version }}
-          version-specifier: ${{ needs.export-version-specifier.outputs.version_specifier }}
-
-  e2e-install2-failing:
-    name: E2E Install2 Failing
-    if: contains(github.event.pull_request.labels.*.name, 'run-failing-install2-tests')
-    runs-on: ${{ matrix.runner || 'ubuntu-22.04' }}
-    needs:
-      - build-current
-      - build-legacydr
-      - build-previous-k0s
-      - build-upgrade
-      - find-previous-stable
-      - release-app
-      - export-version-specifier
-    strategy:
-      fail-fast: false
-      matrix:
-        test:
-          - TestSingleNodeAirgapInstall2Upgrade
-          - TestAlwaysFail
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Download current binary
-        uses: actions/download-artifact@v4
-        with:
-          name: current-release
-          path: output/bin
-
-      - uses: ./.github/actions/e2e
-        with:
-          test-name: '${{ matrix.test }}'
-          is-large-runner: ${{ matrix.runner == 'embedded-cluster' }}
-          airgap-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_LICENSE_ID }}
-          snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE_ID }}
-          snapshot-license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_SNAPSHOT_LICENSE }}
-          airgap-snapshot-license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_AIRGAP_SNAPSHOT_LICENSE_ID }}
-          license-id: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE_ID }}
-          license: ${{ secrets.STAGING_EMBEDDED_CLUSTER_LICENSE }}
-          dr-aws-access-key-id: ${{ secrets.TESTIM_AWS_ACCESS_KEY_ID }}
-          dr-aws-secret-access-key: ${{ secrets.TESTIM_AWS_SECRET_ACCESS_KEY }}
-          k0s-version: ${{ needs.build-current.outputs.k0s_version }}
-          k0s-version-previous: ${{ needs.build-previous-k0s.outputs.k0s_version }}
-          k0s-version-previous-stable: ${{ needs.find-previous-stable.outputs.k0s_version }}
-          version-specifier: ${{ needs.export-version-specifier.outputs.version_specifier }}
-
-
   # this job will validate that all the tests passed
   # it is used for the github branch protection rule
   validate-success:
@@ -1137,10 +907,6 @@ jobs:
       - dryrun-tests
       - check-images
       - check-operator-crds
-      - e2e-docker-install2
-      - e2e-docker-install2-failing
-      - e2e-install2
-      - e2e-install2-failing
     if: always()
     steps:
       # https://docs.github.com/en/actions/learn-github-actions/contexts#needs-context
@@ -1165,17 +931,5 @@ jobs:
       - name: fail if check-operator-crds job was not successful
         if: needs.check-operator-crds.result != 'success'
         run: exit 1
-      - name: fail if e2e-docker-install2 job was not successful
-        if: needs.e2e-docker-install2.result != 'success'
-        run: exit 1
-      - name: fail if e2e-docker-install2-failing job WAS successful
-        if: needs.e2e-docker-install2-failing.result == 'success'
-        run: exit 1
-      - name: fail if e2e-install2 job was not successful
-        if: needs.e2e-install2.result != 'success'
-        run: exit 1
-      - name: fail if e2e-install2-failing job WAS successful
-        if: needs.e2e-install2-failing.result == 'success'
-        run: exit 1
       - name: succeed if everything else passed
         run: echo "Validation succeeded"