Feature/kc 26 (#1369)

* changes to get docker compose to launch cleanly

* fix group issue by upgrading admin client

* fix group issue by upgrading admin client

* refactor keycloak admin client

* try to build with openssl legacy provider

* try to build with openssl legacy provider

* upd all build steps with ssl flag

* fix dockerfile

* fix cypress

* add some output to cypress automation logs

* upd cypress log

* use nightly keycloak and use 2 isntances in docker compose

* use kc-upgrade gwa for cypress e2e

* add data plane info as gwa needs for upstream validation check

* fix group mgmt

* Test api-key plugin with kong 3.9.0

* fix for kong-spec-expose and kong-upstream-jwt

* remove commented out cypress tests

* use our latest Kong 3 docker image

* upgrade gwa-api version in e2e tests workflow

* Fix expected error response for IP restriction

* update gwa-cli to 3.0.6 for e2e

* provide DECK_CLI env var, eliminate docker compose warnings

* replace test host httpbin.org with httpbun.com

* accept 97 to 99

* correct name of template

* Retry if product/dataset fails to publish

* exclude `gwa status` from e2e tests

* Expect 200 when using allowed methods (POST/PUT)

* add retry for `gwa apply` for two-tiered

* Only enable code coverage in CI (GitHub Actions)

* remove it.onlys - oops

* Clear redis rate limit key before new rate limiting test

* move clear redis timing

* upgrade local postgres to 12.8 > 15

* bump actions/cache v2 > v4

* use postgres 14 in e2e

* Changes to get docker compose to run (with authz issues)

* use updated gwa-api, only run cypress tests 01 and 02

* update cypress logout action

* fixes for updated keycloak ui

* fix expected state for scope not found

* run all tests

* test 01 02

* run all tests

* update node version in nvmrc and ci-feat-sonar GHA

* fix decision strategy for default permission

* fix and cleanup keycloak tests

* more test fixes

* bump postgres to 14 in docker compose

* parameterize in docker compose

* set rate limit to 1 for expected one successful call

* add wait before Kong requests after updating auth plugin

* remove echo from local kong dockerfile

* merge consecutive RUN calls

* add Namespace.View on Namespace creation

* Add explicit pagination for Keycloak group retrieval with max 1000

* Update Jest setup with polyfills for Web APIs and updated mocks for Keycloak. Refactor error handling in BatchService.

* adjust depenencies

* fix UMA permission assertion

* remove Namespace.View (return to previous state)

* update SonarCloud Scan action

* remove project from sonar arg

* add api-services-portal project

* restore sonar

* upgrade keycloak-connect

* Replacing local keycloak-admin-client package with official @keycloak/keycloak-admin-client.

* stop copying keycloak-admin-client in dockerfiles

* fix sonar scan action

* code cleanup: use find instead of filter

* code cleanup: remove await on non-promise

* Add test to verify authorization scope enforcement (for ds/api)

* Add ForbiddenError class and update authorization handling to use it for permission denials

* cypress: update expected response code

* cypress: disable documentation tests

* restore cypress to v13

* cypress: add necessary role for Janis for activity tests

* cypress: fix error message

* Cypress: add permissions for Janis in client cred ns

* cypress: force click

* enable keycloak metrics and health

* Cypress: add missing fixture

* cypress: update expected error code to 401 for service account disabled

* cypress: visit admin access page

* Optimize calls to Keycloak

* Fix test for keycloak optimization

* kc26 logout fix

* fix Missing parameters: id_token_hint when logging out

* fix2: Missing parameters: id_token_hint when logging out

* remove backend_logout_url

* Fix redirect to home page, fix signout page typo

* Portal attempt fix to redirect to /signout

* Remove comment that's no longer applicable

* use latest cli

* better handle errors for getOrganization + getOrganizationUnit

* guard gql error if org-unit is null

* fix logout assertion

* Few more merge conflicts, couple remaining

* Another merge conflict, just package-lock remaining

* fix merge conflict package-lock.json, yarn.lock update

* Add skipLibCheck. This is a standard way to avoid dependency type-definition issues and is recommended when using older TypeScript with newer @types packages.

* Fix portal build

* Fix portal build 2

* Fix portal build 3

* Fix portal build 4

* Fix portal build 5

* Fix portal build 6

* Fix portal build 7

* Upgrade babel/core

* Fix sonar tests

* Fix last remaining test

* Fix last remaining test 2

* Fix scope loading issue with cypress

* Fix for 14/04 cypress tests

* accept Namespace.Manage or View for /activity endpoints

* use gwa-api `dev` branch for e2e

* kc bump + minor tweaks

---------

Co-authored-by: James Elson <james.elson@gov.bc.ca>
Co-authored-by: Russell Vinegar <russell.vinegar@gov.bc.ca>

Author: 3 people

.dockerignore

@@ -1,9 +1,10 @@
-src/node_modules
 src/.npm
 src/_tmp
 src/.cache
 src/.config
 src/.nyc_output
 src/dist
 **/.next
-_data
+_data
+**/node_modules
+__coverage__

.env.local

@@ -29,6 +29,7 @@ EMAIL_USER=
 EMAIL_PASS=
 EXTERNAL_URL=http://oauth2proxy.localtest.me:4180
 OIDC_ISSUER=http://keycloak.localtest.me:9081/auth/realms/master
+OIDC_CLIENT_ID=aps-portal
 LOCAL_ENV=true
 WORKING_PATH=/tmp
 DESTINATION_URL=

.github/workflows/aps-cypress-e2e.yaml

@@ -25,7 +25,7 @@ jobs:
     steps:
       - name: Build GWA API Image
         run: |
-          git clone https://github.com/bcgov/gwa-api.git --branch v1.0.47
+          git clone https://github.com/bcgov/gwa-api.git --branch dev
           cd gwa-api/microservices/gatewayApi
           docker build -t gwa-api:e2e .
 

.github/workflows/ci-build-deploy.yaml

@@ -54,7 +54,7 @@ jobs:
           username: ${{ env.REGISTRY_USERNAME }}
           password: ${{ env.REGISTRY_PASSWORD }}
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: /tmp/.buildx-cache
           key: ${{ runner.os }}-buildx-${{ github.sha }}
@@ -318,6 +318,9 @@ jobs:
                     - name: X-Forwarded-Access-Token
                       values:
                         - claim: access_token
+                    - name: X-Forwarded-Id-Token
+                      values:
+                        - claim: IDToken
                   injectResponseHeaders: []
                   metricsServer:
                     BindAddress: ""
@@ -394,6 +397,8 @@ jobs:
               secure: true
             OIDC_ISSUER:  
               value: '${{ secrets.OIDC_ISSUER }}'
+            OIDC_CLIENT_ID:
+              value: '${{ secrets.OIDC_CLIENT_ID }}'
             JWKS_URL:
               value: '${{ secrets.OIDC_ISSUER }}/protocol/openid-connect/certs'
             EXTERNAL_URL:

.github/workflows/ci-build-feeders.yaml

@@ -38,7 +38,7 @@ jobs:
           username: ${{ env.REGISTRY_USERNAME }}
           password: ${{ env.REGISTRY_PASSWORD }}
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: /tmp/.buildx-cache
           key: ${{ runner.os }}-buildx-${{ github.sha }}

.github/workflows/ci-build-only.yaml

@@ -46,7 +46,7 @@ jobs:
           username: ${{ env.REGISTRY_USERNAME }}
           password: ${{ env.REGISTRY_PASSWORD }}
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: /tmp/.buildx-cache
           key: ${{ runner.os }}-buildx-${{ github.sha }}

.github/workflows/ci-feat-sonar.yaml

@@ -48,7 +48,14 @@ jobs:
           docker compose down
 
       - name: SonarCloud Scan
-        uses: sonarsource/sonarqube-scan-action@master
+        uses: sonarsource/sonarqube-scan-action@v6
+        with:
+          args: >
+            -Dsonar.organization=bcgov-sonarcloud
+            -Dsonar.projectKey=bcgov_api-services-portal
+            -Dsonar.sources=src/auth,src/authz,src/batch,src/services
+            -Dsonar.javascript.lcov.reportPaths=./src/__coverage__/lcov.info
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+          SONAR_HOST_URL: https://sonarcloud.io

.github/workflows/main.yaml

@@ -44,7 +44,7 @@ jobs:
           echo "::set-output name=APP_REVISION::${GITHUB_SHA}"
         id: set-deploy-id
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
           path: /tmp/.buildx-cache
           key: ${{ runner.os }}-buildx-${{ github.sha }}

.nvmrc

@@ -1 +1 @@
-v16.15.1
+v22.21.1

docker-compose.yml

@@ -11,39 +11,52 @@ secrets:
 
 services:
   keycloak:
-    image: quay.io/keycloak/keycloak:15.1.1
+    image: keycloak-quarkus:e2e
     container_name: keycloak
-    hostname: keycloak
+    build:
+      context: local/keycloak
+      dockerfile: Dockerfile
+      args:
+        - KC_VERSION=${KC_VERSION:-26.5.3}
     depends_on:
       kong-db:
         condition: service_healthy
+    environment:
+      KC_BOOTSTRAP_ADMIN_USERNAME: admin
+      KC_BOOTSTRAP_ADMIN_PASSWORD: local
+    # Keycloak 26 Hostname v2: --hostname accepts full URL (host + port + path).
     command:
       [
-        '-b',
-        '0.0.0.0',
-        '-Djboss.socket.binding.port-offset=1001',
-        '-Dkeycloak.migration.action=import',
-        '-Dkeycloak.migration.provider=singleFile',
-        '-Dkeycloak.migration.file=/tmp/realm-config/master-realm.json',
-        '-Dkeycloak.migration.strategy=OVERWRITE_EXISTING',
-        '-Dkeycloak.profile.feature.upload_scripts=enabled',
+        "start",
+        "--optimized",
+        "--hostname",
+        "http://keycloak.localtest.me:9081/auth",
+        "--db-url",
+        "jdbc:postgresql://kong-db/${KEYCLOAK_DB_NAME:-keycloak}",
+        "--db-username",
+        "keycloakuser",
+        "--db-password",
+        "keycloakuser",
+        "--http-enabled",
+        "true",
+        "--http-port",
+        "9081",
+        "--import-realm",
       ]
-    ports:
-      - 9081:9081/tcp
-    environment:
-      #KEYCLOAK_USER: local
-      #KEYCLOAK_PASSWORD: local
-      DB_VENDOR: POSTGRES
-      DB_SCHEMA: public
-      DB_ADDR: kong-db:5432
-      DB_USER: keycloakuser
-      DB_PASSWORD: keycloakuser
     volumes:
-      - ./local/keycloak/master-realm.json:/tmp/realm-config/master-realm.json
+      - ./local/keycloak/master-realm.json:/opt/keycloak/data/import/master-realm.json
+    ports:
+      - 9081:9081
+      - 9000:9000
     networks:
       aps-net:
         aliases:
           - keycloak.localtest.me
+    healthcheck:
+      test: timeout 10s bash -c ':> /dev/tcp/localhost/9081'
+      interval: 5s
+      timeout: 5s
+      retries: 5
   oauth2-proxy:
     image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
     container_name: oauth2-proxy
@@ -62,7 +75,7 @@ services:
           - oauth2proxy.localtest.me
   apsportal:
     container_name: apsportal
-    image: apsportal:latest
+    image: apsportal:${APSPORTAL_TAG:-latest}
     depends_on:
       - keycloak
     build:
@@ -198,7 +211,7 @@ services:
         aliases:
           - redis-master
   gwa-api:
-    image: gwa-api:e2e
+    image: gwa-api:${GWA_API_TAG:-e2e}
     container_name: gwa-api
     entrypoint: sh -c "chmod +x /tmp/gwa/entrypoint.sh && sh /tmp/gwa/entrypoint.sh"
     ports: