empower-registry-testing (#514)

Summary:

- Dynamically generate `nginx` LB config.
- Expand LB testing.
- Improved documentation for reverse proxied testing.

Author: general-kroll-4-life

.github/workflows/build.yml

@@ -492,7 +492,8 @@ jobs:
         } >> "${GITHUB_ENV}"
         
         if [ "${{ matrix.registry }}" = "test/registry" ]; then
-          echo '127.0.0.1   storage.googleapis.com' | sudo tee -a /etc/hosts
+          python3 test/python/tcp_lb.py --generate-hosts-entries | sudo tee -a /etc/hosts
+          python3 test/python/tcp_lb.py --generate-nginx-lb > test/tcp/reverse-proxy/nginx/dynamic-sni-proxy.conf
         fi
 
 
@@ -510,7 +511,7 @@ jobs:
                   | sudo tee /etc/apt/sources.list.d/nginx.list
             sudo apt-get update
             sudo apt-get install nginx
-            sudo nginx -c "$(pwd)/test/tcp/reverse-proxy/nginx/elegant-sni-proxy.conf"
+            sudo nginx -c "$(pwd)/test/tcp/reverse-proxy/nginx/dynamic-sni-proxy.conf"
           fi
 
     - name: Install Python dependencies

docs/CICD.md

@@ -13,9 +13,7 @@ Summary:
     - This pattern does the below:
         - (a) Build and push by digest.
         - (b) Leverage [`docker buildx imagetools`](https://docs.docker.com/reference/cli/docker/buildx/imagetools/) to write desired tags.
-    - This pattern is only required because if tag pushes are done concurrently, then identical multi-architecture tags are clobbered in a reverse race condition. 
-    - **NOTE**: The QEMU build for linux/arm64 is **very slow**.  On the order of 30 minutes.  This is currently unavoidable.
-    - **TODO**: Migrate linux/arm64 docker build to native once GHA supports this platform as a first class citizen.
+    - This pattern is only required because if tag pushes are done concurrently, then identical multi-architecture tags are clobbered in a reverse race condition.
     - **NOTE**.  Be careful selecting from [the available github actions runners](https://github.com/actions/runner-images).  Some runnner instances may be incompatible with assumed toolchain.
 - Docker images:
     - [production image at `stackql/stackql`](https://hub.docker.com/r/stackql/stackql/tags).

test/python/tcp_lb.py

@@ -0,0 +1,210 @@
+import argparse
+
+from typing import List, Tuple, Iterable, Optional
+
+_DEFAULT_PORT = 1070
+_GOOGLE_DEFAULT_PORT = 1080
+_GOOGLEADMIN_DEFAULT_PORT = 1098
+_OKTA_DEFAULT_PORT = 1090
+_AWS_DEFAULT_PORT = 1091
+_K8S_DEFAULT_PORT = 1092
+_GITHUB_DEFAULT_PORT = 1093
+_AZURE_DEFAULT_PORT = 1095
+_SUMOLOGIC_DEFAULT_PORT = 1096
+_DIGITALOCEAN_DEFAULT_PORT = 1097
+_STACKQL_TEST_DEFAULT_PORT = 1099
+_STACKQL_AUTH_TESTING_DEFAULT_PORT = 1170
+
+_DEFAULT_AWS_GLOBAL_SERVICES: Tuple[str] = (
+    'iam',
+    'route53',
+)
+
+_DEFAULT_AWS_REGIONAL_SERVICES: Tuple[str] = (
+    "cloudcontrolapi", # cloudcontrol
+    'cloudhsmv2', # cloudhsm
+    "ec2",
+    'logs', # cloudwatch
+    "s3",
+    "transfer",
+)
+
+_DEFAULT_AWS_REGIONS: Tuple[str] = (
+    "us-east-1",
+    "us-east-2",
+    "us-west-1",
+    "us-west-2",
+    "ap-south-1",
+    "ap-northeast-1",
+    "ap-northeast-2",
+    "ap-southeast-1",
+    "ap-southeast-2",
+    "ap-northeast-3",
+    "ca-central-1",
+    "eu-central-1",
+    "eu-west-1",
+    "eu-west-2",
+    "eu-west-3",
+    "eu-north-1",
+    "sa-east-1",
+)
+
+_DEFAULT_GCP_SERVICES: Tuple[str] = (
+    'compute',
+    'storage',
+)
+
+class _LB(object):
+
+    def __init__(self, lb_host: str, lb_port, backend_host: str, backend_port: int) -> None:
+        self._lb_host = lb_host
+        self._lb_port = lb_port
+        self._backend_host = backend_host
+        self._backend_port = backend_port
+    
+    def get_lb_host(self) -> str:
+        return self._lb_host
+    
+    def get_lb_port(self) -> int:
+        return self._lb_port
+    
+    def get_backend_host(self) -> str:
+        return self._backend_host
+    
+    def get_backend_port(self) -> int:
+        return self._backend_port
+
+class _HostsGenerator(object):
+
+    def __init__(
+            self, 
+            aws_global_services: Optional[Iterable[str]] = None, 
+            aws_regional_services: Optional[Iterable[str]] = None, 
+            aws_regions: Optional[Iterable[str]] = None, 
+            google_services: Optional[Iterable[str]] = None
+        ) -> None:
+        self._aws_global_services = aws_global_services if aws_global_services is not None else _DEFAULT_AWS_GLOBAL_SERVICES
+        self._aws_regional_services = aws_regional_services if aws_regional_services is not None else _DEFAULT_AWS_REGIONAL_SERVICES
+        self._aws_regions = aws_regions if aws_regions is not None else _DEFAULT_AWS_REGIONS
+        self._google_services = google_services if google_services is not None else _DEFAULT_GCP_SERVICES
+
+    @staticmethod
+    def _generate_aws_hosts(aws_regional_services: Iterable[str], aws_regions: Iterable[str], aws_global_services: Iterable[str]) -> Iterable[_LB]:
+        for service in aws_regional_services:
+            for region in aws_regions:
+                yield _LB(f'{service}.{region}.amazonaws.com', 443, '127.0.0.1', _AWS_DEFAULT_PORT)
+        for service in aws_global_services:
+            yield _LB(f'{service}.amazonaws.com', 443, '127.0.0.1', _AWS_DEFAULT_PORT)
+
+    @staticmethod
+    def _generate_gcp_hosts(google_services: Iterable[str]) -> Iterable[_LB]:
+        for service in google_services:
+            yield _LB(f'{service}.googleapis.com', 443, '127.0.0.1', _GOOGLE_DEFAULT_PORT)
+        
+    def generate_all_load_balancers(self) -> Iterable[_LB]:
+        yield from self._generate_aws_hosts(self._aws_regional_services, self._aws_regions, self._aws_global_services)
+        yield from self._generate_gcp_hosts(self._google_services)
+
+
+class _NginxConfigGenerator(object):
+
+    def __init__(self, n_indent: int = 2) -> None:
+        self._n_indent = n_indent
+
+    @staticmethod
+    def _generate_backends(hosts: Iterable[_LB], indent: int) -> Iterable[str]:
+        """
+        Generate the nginx config.
+        """
+        return [
+            " " * indent + f'{host.get_lb_host()}    {host.get_backend_host()}:{host.get_backend_port()};'
+            for host in hosts
+        ]
+    
+    def _generate_file_content(self, hosts: Iterable[_LB]) -> Tuple[str]:
+        """
+        Generate the file content.
+        """
+        lines : Tuple[str] = (
+            'worker_processes  1;',
+            '',
+            '#error_log  logs/error.log;',
+            '#error_log  logs/error.log  notice;',
+            '#error_log  logs/error.log  info;',
+            '',
+            '#pid        logs/nginx.pid;',
+            '',
+            '',
+            'events {',
+            f'{" " * self._n_indent}worker_connections  1024;',
+            '}',
+            '',
+            'stream {',
+            '',
+            f'{" " * self._n_indent}map $ssl_preread_server_name $targetBackend {{',
+            *self._generate_backends(hosts, self._n_indent * 2),
+            f'{" " * self._n_indent}}}',
+            '',
+            f'{" " * self._n_indent}server {{',
+            f'{" " * self._n_indent}listen 443;',
+            f'{" " * self._n_indent}',
+            f'{" " * self._n_indent}proxy_connect_timeout 1s;',
+            f'{" " * self._n_indent}proxy_timeout 3s;',
+            f'{" " * self._n_indent}resolver 1.1.1.1;',
+            f'{" " * self._n_indent}',
+            f'{" " * self._n_indent}proxy_pass $targetBackend;',
+            f'{" " * self._n_indent}ssl_preread on;',
+            f'{" " * self._n_indent}}}',
+            '}',
+        )
+        return lines
+
+    def generate_file_content(self, hosts: Iterable[_LB]) -> Tuple[str]:
+        """
+        Generate the file content.
+        """
+        return self._generate_file_content(hosts)
+
+
+class _HostsFileEntriesGenerator(object):
+
+    def __init__(self, hosts: Iterable[_LB]) -> None:
+        self._hosts = hosts
+
+    def generate_entries(self) -> Iterable[str]:
+        """
+        Generate the entries.
+        """
+        return tuple(
+            f'{host.get_backend_host()}    {host.get_lb_host()}'
+            for host in self._hosts
+        )
+
+
+def _parse_args() -> argparse.Namespace:
+    """
+    Parse the arguments.
+    """
+    parser = argparse.ArgumentParser(description='Create a token.')
+    parser.add_argument('--generate-nginx-lb', help='Opt-in nginx config generation', action=argparse.BooleanOptionalAction)
+    parser.add_argument('--generate-hosts-entries', help='Opt-in hosts files entries generation', action=argparse.BooleanOptionalAction)
+    # parser.add_argument('--header', type=str, help='The header.')
+    return parser.parse_args()
+
+def main():
+    args = _parse_args()
+    host_gen = _HostsGenerator()
+    all_hosts = [lb for lb in host_gen.generate_all_load_balancers()]
+    nginx_cfg_gen = _NginxConfigGenerator()
+    if args.generate_nginx_lb:
+        for l in nginx_cfg_gen.generate_file_content(all_hosts):
+            print(l)
+        return
+    if args.generate_hosts_entries:
+        hosts_gen = _HostsFileEntriesGenerator(all_hosts)
+        for l in hosts_gen.generate_entries():
+            print(l)
+        return
+
+if __name__ == '__main__':
+    main()

test/robot/functional/stackql_mocked_from_cmd_line.robot

@@ -7321,6 +7321,7 @@ Set Statement Update Auth Scenario Working
     ...    stderr=${CURDIR}/tmp/Set-Statement-Update-Auth-Scenario-Working-Working-stderr.tmp
 
 Busted Auth Throws Error Then Set Statement Update Auth Scenario Working
+    [Tags]    registry    tls_proxied
     ${inputStr} =    Catenate
     ...    select name, id, network, split_part(network, '/', 8) as network_region from google.compute.firewalls where project \= 'testing-project' order by id desc;
     ...    set session "$.auth.google.credentialsfilepath"='${AUTH_GOOGLE_SA_KEY_PATH}';
@@ -7395,6 +7396,7 @@ Alternate App Root Persists All Temp Materials in Alotted Directory
     Directory Should Exist    ${TEST_TMP_EXEC_APP_ROOT_NATIVE}${/}src
 
 View Tuple Replacement Working As Exemplified by AWS EC2 Instances List and Detail
+    [Tags]    registry    tls_proxied
     ${inputStr} =    Catenate
     ...    SELECT region, instance_id, tenancy, security_groups 
     ...    FROM aws.ec2_nextgen.instances 
@@ -7427,7 +7429,7 @@ View Tuple Replacement Working As Exemplified by AWS EC2 Instances List and Deta
     ...    repeat_count=20
 
 Google Buckets List With Date Logic Exemplifies Use of SQLite Math Functions
-    [Tags]   registry    tls_proxied
+    [Tags]    registry    tls_proxied
     Pass Execution If    "${SQL_BACKEND}" == "postgres_tcp"    This is a valid case where the test is targetted at SQLite only
     ${inputStr} =    Catenate
     ...    SELECT name, timeCreated, floor(julianday('2025-01-27')-julianday(timeCreated)) as days_since_ceiling 

test/tcp/reverse-proxy/nginx/.gitignore

@@ -0,0 +1 @@
+dynamic-sni-proxy.conf

test/tcp/reverse-proxy/nginx/README.md

@@ -2,14 +2,16 @@
 
 ## Runnning nginx as a pass through TCP proxy
 
+These scenarios require `python` >= `3.11`, `nginx` >= `1.27.3` (and all of the `pip` requirements of this repository).
 
-### Automated testing scenario
+This is not ideal for local running; great for CI lifecycles.  If you do this locally, you **must** restore `/etc/hosts`.  
+Rough and ready `/etc/hosts` restoration instructions are supplied, but, honestly, it is better if you take a prior backup and 
+clobber with same when done.
 
-This scenario requires `nginx` >= version `1.27.3` (and all of the `pip` requirements of this repository).
+We will need superuser privileges at times (via `sudo`) because we are editing `/etc/hosts` and using the protected port `443`.
 
-This is not ideal for local running; great for CI lifecycles.  If you do this locally, you **must** restore `/etc/hosts`.
+### Automated testing scenario
 
-We will need superuser privileges at times (via `sudo`) because we are editing `/etc/hosts` and using the protected port `443`.
 
 Please run all commands from the root of the repository:
 
@@ -41,3 +43,39 @@ robot \
 sed '/storage.googleapis.com/d' /etc/hosts | sudo tee /etc/hosts
 
 ```
+
+### More fulsome testing
+
+Please run all commands from the root of the repository:
+
+```bash
+
+python test/python/tcp_lb.py --generate-hosts-entries | sudo tee -a /etc/hosts
+
+python test/python/tcp_lb.py --generate-nginx-lb > test/tcp/reverse-proxy/nginx/dynamic-sni-proxy.conf
+
+sudo nginx -c $(pwd)/test/tcp/reverse-proxy/nginx/tls-pass-through.conf
+
+```
+
+Then, run the robot tests that are suitable (as tags evolve, you may want to be more selective):
+
+```bash
+
+robot \
+  --variable 'SUNDRY_CONFIG:{"registry_path": "test/registry"}' \
+  --include tls_proxied \
+  -d test/robot/reports \
+  test/robot/functional
+
+```
+
+**Very important**; after this, restore `/etc/hosts` (BSD and GNU compatible, clunky expression):
+
+```bash
+
+sed '/googleapis/d' /etc/hosts | sudo tee /etc/hosts
+
+sed '/aws/d' /etc/hosts | sudo tee /etc/hosts
+
+```