Skip to content

vpc-live-testing #302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
general-kroll-4-life merged 2 commits into from
Apr 14, 2025
Merged

vpc-live-testing #302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b076de3
vpc-lifecycle-testing
general-kroll-4-life Apr 14, 2025
4fe2b84
Merge pull request #301 from stackql/feature/vpc-testing
general-kroll-4-life Apr 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,12 +241,12 @@ jobs:
if: success()
id: live_integration_tests_readonly
env:
GOOGLE_CREDENTIALS: ${{ secrets.CI_SCENARIO_GCP_RO_SECRET }}
GOOGLE_CREDENTIALS: ${{ secrets.CI_ROBOT_GCP_RO_SECRET }}
AWS_ACCESS_KEY_ID: ${{ secrets.CI_SCENARIO_RO_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.CI_SCENARIO_RO_AWS_SECRET_ACCESS_KEY }}
run: |
providerRoot="$(realpath $(pwd)/providers)"
sundryCfg='SUNDRY_CONFIG:{"registry_path": "'"${providerRoot}"'", "GCS_BUCKET_NAME": "stackql-demo-bucket-02", "GCP_PROJECT": "stackql-demo", "AWS_RECORD_SET_ID": "A00000001AAAAAAAAAAAA", "AWS_RECORD_SET_REGION": "us-east-1"}'
sundryCfg='SUNDRY_CONFIG:{"registry_path": "'"${providerRoot}"'", "GCS_BUCKET_NAME": "stackql-robot-bucket-02", "GCP_PROJECT": "stackql-robot", "AWS_RECORD_SET_ID": "A00000001AAAAAAAAAAAA", "AWS_RECORD_SET_REGION": "us-east-1"}'
robot \
--variable "${sundryCfg}" \
--variable SHOULD_RUN_DOCKER_EXTERNAL_TESTS:true \
Expand All @@ -270,12 +270,12 @@ jobs:
if: github.ref_type == 'tag'
id: live_integration_tests_readwrite
env:
GOOGLE_CREDENTIALS: ${{ secrets.CI_SCENARIO_GCP_RW_SECRET }}
GOOGLE_CREDENTIALS: ${{ secrets.CI_ROBOT_GCP_RW_SECRET }}
AWS_ACCESS_KEY_ID: ${{ secrets.CI_SCENARIO_RW_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.CI_SCENARIO_RW_AWS_SECRET_ACCESS_KEY }}
run: |
providerRoot="$(realpath $(pwd)/providers)"
sundryCfg='SUNDRY_CONFIG:{"registry_path": "'"${providerRoot}"'", "GCS_BUCKET_NAME": "stackql-demo-bucket-02", "GCP_PROJECT": "stackql-demo", "AWS_RECORD_SET_ID": "A00000001AAAAAAAAAAAA", "AWS_RECORD_SET_REGION": "us-east-1"}'
sundryCfg='SUNDRY_CONFIG:{"registry_path": "'"${providerRoot}"'", "GCS_BUCKET_NAME": "stackql-robot-bucket-02", "GCP_PROJECT": "stackql-robot", "AWS_RECORD_SET_ID": "A00000001AAAAAAAAAAAA", "AWS_RECORD_SET_REGION": "us-east-1"}'
robot \
--variable "${sundryCfg}" \
--variable SHOULD_RUN_DOCKER_EXTERNAL_TESTS:true \
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ stackql-zip
stackql-core/
stackql-any-sdk/
.venv/
*.py[co]
Binary file removed test/robot/stackql/live/readonly/__pycache__/readonly_variables.cpython-313.pyc
View file
Open in desktop
Binary file not shown.
10 changes: 5 additions & 5 deletions test/robot/stackql/live/readonly/readonly_variables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ def _get_expected_gcs_bucket_check(
Expected GCS bucket check.
"""
return '' + \
'|------------------------|----------------|----------------|\n' + \
'| name | softDeleteTime | hardDeleteTime |\n' + \
'|------------------------|----------------|----------------|\n' + \
'| stackql-demo-bucket-01 | null | null |\n' + \
'|------------------------|----------------|----------------|'
'|-------------------------|----------------|----------------|\n' + \
'| name | softDeleteTime | hardDeleteTime |\n' + \
'|-------------------------|----------------|----------------|\n' + \
'| stackql-robot-bucket-01 | null | null |\n' + \
'|-------------------------|----------------|----------------|'


def get_variables(
Expand Down
4 changes: 2 additions & 2 deletions test/robot/stackql/live/readonly/stackql.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ ${USE_STACKQL_PREINSTALLED} false # to be overridden from command l
${CORE_PREFIX} stackql-core
${SUNDRY_CONFIG}= SEPARATOR=
... {
... "GCS_BUCKET_NAME": "stackql-demo-bucket-01",
... "GCP_PROJECT": "stackql-demo",
... "GCS_BUCKET_NAME": "stackql-robot-bucket-01",
... "GCP_PROJECT": "stackql-robot",
... "AWS_RECORD_SET_ID": "A00000001AAAAAAAAAAAA",
... "AWS_RECORD_SET_REGION": "us-east-1",
... "registry_path": "${CURDIR}${/}..${/}..${/}..${/}..${/}..${/}providers"
Expand Down
Binary file removed test/robot/stackql/live/readwrite/__pycache__/readwrite_variables.cpython-313.pyc
View file
Open in desktop
Binary file not shown.
118 changes: 118 additions & 0 deletions test/robot/stackql/live/readwrite/google_networks_lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@




```sql

--
insert into google.compute.networks(data__name, data__autoCreateSubnetworks, project)
select 'robot-vpc-01', false, 'stackql-robot';
The operation was despatched successfully

--
select id, name, description, subnetworks from google.compute.networks where name = 'robot-vpc-01' and project = 'stackql-robot';
|--------------------|--------------|-------------|-------------|
| id | name | description | subnetworks |
|--------------------|--------------|-------------|-------------|
| 987158103920671616 | robot-vpc-01 | null | null |
|--------------------|--------------|-------------|-------------|

--
insert into google.compute.subnetworks(data__name, data__ipCidrRange, data__description, data__network, project, region)
select
'robot-subnet-01',
'10.0.0.0/8',
'An immutable ROBOT subnet description.',
'projects/stackql-robot/global/networks/robot-vpc-01',
'stackql-robot',
'australia-southeast1'
;
The operation was despatched successfully

---
select id, name, description, subnetworks from google.compute.networks where name = 'robot-vpc-01' and project = 'stackql-robot';
|--------------------|--------------|-------------|---------------------------------------------------------------------------------------------------------------------------|
| id | name | description | subnetworks |
|--------------------|--------------|-------------|---------------------------------------------------------------------------------------------------------------------------|
| 987158103920671616 | robot-vpc-01 | null | ["https://www.googleapis.com/compute/v1/projects/stackql-robot/regions/australia-southeast1/subnetworks/robot-subnet-01"] |
|--------------------|--------------|-------------|---------------------------------------------------------------------------------------------------------------------------|

--
select name, id, secondaryIpRanges, fingerprint from google.compute.subnetworks where name = 'robot-subnet-01' and region = 'australia-southeast1' and project = 'stackql-robot' ;
|-----------------|---------------------|-------------------|--------------|
| name | id | secondaryIpRanges | fingerprint |
|-----------------|---------------------|-------------------|--------------|
| robot-subnet-01 | 7691273977604797678 | null | UNiA_jWveJI= |
|-----------------|---------------------|-------------------|--------------|


---
update google.compute.subnetworks
SET
data__secondaryIpRanges = '[
{
"ipCidrRange": "192.168.0.0/24",
"rangeName": "robot-range-01"
}
]',
data__fingerprint = 'UNiA_jWveJI='
WHERE
project = 'stackql-robot'
and
region = 'australia-southeast1'
and
subnetwork = 'robot-subnet-01'
;
The operation was despatched successfully

--
select name, id, secondaryIpRanges, fingerprint from google.compute.subnetworks where name = 'robot-subnet-01' and region = 'australia-southeast1' and project = 'stackql-robot' ;
|-----------------|---------------------|-----------------------------------------------------------------|--------------|
| name | id | secondaryIpRanges | fingerprint |
|-----------------|---------------------|-----------------------------------------------------------------|--------------|
| robot-subnet-01 | 7691273977604797678 | [{"ipCidrRange":"192.168.0.0/24","rangeName":"robot-range-01"}] | vmdP1iA6Cfc= |
|-----------------|---------------------|-----------------------------------------------------------------|--------------|


--
delete from
google.compute.subnetworks
WHERE
project = 'stackql-robot'
and
region = 'australia-southeast1'
and
subnetwork = 'robot-subnet-01'
;
<nil displayed response>

--
select name, id, secondaryIpRanges, fingerprint from google.compute.subnetworks where name = 'robot-subnet-01' and region = 'australia-southeast1' and project = 'stackql-robot' ;
|------|----|-------------------|-------------|
| name | id | secondaryIpRanges | fingerprint |
|------|----|-------------------|-------------|


--
delete from
google.compute.networks
WHERE
project = 'stackql-robot'
and
network = 'robot-vpc-01'
;
<nil displayed response>


--
select id, name, description, subnetworks from google.compute.networks where name = 'robot-vpc-01' and project = 'stackql-robot';
|----|------|-------------|-------------|
| id | name | description | subnetworks |
|----|------|-------------|-------------|

```

## Notes

- Even compute api is not on by default in a new google project.

200 changes: 200 additions & 0 deletions test/robot/stackql/live/readwrite/google_readwrite.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
*** Settings ***
Resource ${CURDIR}/stackql.resource
Test Teardown Stackql Per Test Teardown

*** Test Cases ***
Google Buckets Lifecycle
[Documentation] This test case inserts a bucket ("row") into the google.storage.buckets "table", checks the row was inserted, deletes the row, and checks the row was deleted.
[Tags] google storage buckets gooogle.storage google.storage.buckets
${insertInputStr} = Catenate
... insert into google.storage.buckets(data__name, project)
... select '${GCS_BUCKET_NAME}', '${GCP_PROJECT}';
${checkInputStr} = Catenate
... select name, "softDeleteTime", "hardDeleteTime" from google.storage.buckets where bucket = '${GCS_BUCKET_NAME}';
${deleteInputStr} = Catenate
... delete from google.storage.buckets where bucket = '${GCS_BUCKET_NAME}';
Stock Stackql Exec Inline Equals Both Streams
... ${insertInputStr}
... ${EMPTY}
... The operation was despatched successfully
... Google-Buckets-Lifecycle-Insert
Stock Stackql Exec Inline Equals Both Streams
... ${checkInputStr}
... ${EXPECTED_GCS_BUCKET_CHECK}
... ${EMPTY}
... Google-Buckets-Lifecycle-Post-Insert-Check
Stock Stackql Exec Inline Equals Both Streams
... ${deleteInputStr}
... ${EMPTY}
... The operation was despatched successfully
... Google-Buckets-Lifecycle-Delete
Sleep 5s
Stock Stackql Exec Inline Equals Stdout
... ${checkInputStr}
... ${EXPECTED_EMPTY_GCS_BUCKET_CHECK}
... Google-Buckets-Lifecycle-Post-Delete-Check

Google Network Lifecycle
[Documentation] This test case inserts a VPC network then a subnet then mutates the subnet then deletes it and then deletes the network.
[Tags] google compute networks subnetworks vpc gooogle.compute google.compute.networks google.compute.subnetworks Google${SPACE}Networks
${insertNetworkInputStr} = Catenate
... insert into google.compute.networks(data__name, data__autoCreateSubnetworks, project)
... select 'robot-vpc-01', false, '${GCP_PROJECT}';
${insertSubnetInputStr} = Catenate
... insert into google.compute.subnetworks(
... data__name,
... data__ipCidrRange,
... data__description,
... data__network,
... project,
... region)
... select
... 'robot-subnet-01',
... '10.0.0.0/8',
... 'An immutable subnet description.',
... 'projects/${GCP_PROJECT}/global/networks/robot-vpc-01',
... '${GCP_PROJECT}',
... 'australia-southeast1'
... ;
${getNetworkDetailQueryStr} = Catenate
... select name, description, subnetworks
... from google.compute.networks
... where
... name = 'robot-vpc-01'
... and project = '${GCP_PROJECT}';
${getSubnetDetailQueryStr} = Catenate
... select name, secondaryIpRanges, fingerprint
... from google.compute.subnetworks
... where
... name = 'robot-subnet-01'
... and region = 'australia-southeast1'
... and project = '${GCP_PROJECT}'
... ;
${insertNetworkInputStr} = Catenate
... insert into google.compute.networks(data__name, data__autoCreateSubnetworks, project)
... select 'robot-vpc-01', false, '${GCP_PROJECT}';
${deleteNetworkStr} = Catenate
... delete from google.compute.networks where network = 'robot-vpc-01' and project = '${GCP_PROJECT}';
${deleteSubnetStr} = Catenate
... delete from google.compute.subnetworks where subnetwork = 'robot-subnet-01' and project = '${GCP_PROJECT}' and region = 'australia-southeast1';
Stock Stackql Exec Inline Equals Both Streams
... ${insertNetworkInputStr}
... ${EMPTY}
... The operation was despatched successfully
... Google-Network-Lifecycle-Insert-Network
Sleep 20s
${networkResult} = Catenate SEPARATOR=\n
... |--------------|-------------|-------------|
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}description${SPACE}|${SPACE}subnetworks${SPACE}|
... |--------------|-------------|-------------|
... |${SPACE}robot-vpc-01${SPACE}|${SPACE}null${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}null${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
... |--------------|-------------|-------------|
Stock Stackql Exec Inline Equals Both Streams
... ${getNetworkDetailQueryStr}
... ${networkResult}
... ${EMPTY}
... Google-Network-Lifecycle-Select-Network-01
Sleep 20s
Stock Stackql Exec Inline Equals Both Streams
... ${insertSubnetInputStr}
... ${EMPTY}
... The operation was despatched successfully
... Google-Network-Lifecycle-Insert-Subnet
Sleep 20s
${networkDetailResult} = Run Process
... ${STACKQL_EXE}
... \-\-output\=json
... \-\-registry
... ${REGISTRY_LOCAL_NO_VERIFY_CFG_STR}
... exec
... ${getNetworkDetailQueryStr}
... stdout=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-network-detail-1.tmp
... stderr=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-network-detail-1-stderr.tmp
Log Detail dict string = ${networkDetailResult.stdout}
${networkDetailDictList} = Evaluate json.loads($networkDetailResult.stdout)
Log Detail is:${networkDetailDictList}[0]
Should Contain ${networkDetailDictList[0]["subnetworks"]} robot\-subnet\-01
${detailResult} = Run Process
... ${STACKQL_EXE}
... \-\-output\=json
... \-\-registry
... ${REGISTRY_LOCAL_NO_VERIFY_CFG_STR}
... exec
... ${getSubnetDetailQueryStr}
... stdout=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-subnet-detail-1.tmp
... stderr=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-subnet-detail-1-stderr.tmp
Log Detail dict string = ${detailResult.stdout}
${detailDict} = Evaluate json.loads($detailResult.stdout)
Should Be Equal ${detailDict[0]["name"]} robot\-subnet\-01
${CAPTURED_FINGERPRINT} = Set Variable ${detailDict[0]["fingerprint"]}
${updateSubnetInputStr} = Catenate
... update google.compute.subnetworks
... SET
... data__secondaryIpRanges = '[
... {
... "ipCidrRange": "192.168.0.0/24",
... "rangeName": "s-r-01"
... }
... ]',
... data__fingerprint = '${CAPTURED_FINGERPRINT}'
... WHERE
... project = '${GCP_PROJECT}'
... and
... region = 'australia-southeast1'
... and
... subnetwork = 'robot-subnet-01'
... ;
Stock Stackql Exec Inline Equals Both Streams
... ${updateSubnetInputStr}
... ${EMPTY}
... The operation was despatched successfully
... Google-Buckets-Lifecycle-Delete
Sleep 30s
${updatedDetailResult} = Run Process
... ${STACKQL_EXE}
... \-\-output\=json
... \-\-registry
... ${REGISTRY_LOCAL_NO_VERIFY_CFG_STR}
... exec
... ${getSubnetDetailQueryStr}
... stdout=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-subnet-detail-2.tmp
... stderr=${CURDIR}${/}tmp${/}Google-Network-Lifecycle-get-subnet-detail-2-stderr.tmp
Log Updated detail dict string = ${updatedDetailResult.stdout}
${updatedDetailDict} = Evaluate json.loads($updatedDetailResult.stdout)
Should Contain ${updatedDetailDict[0]["secondaryIpRanges"]} 192.168.0.0/24
Sleep 30s
Stock Stackql Exec Inline Equals Both Streams
... ${deleteSubnetStr}
... ${EMPTY}
... ${EMPTY}
... Google-Network-Lifecycle-Subnet-Delete
Sleep 20s
${networkZeroSubnetResult} = Catenate SEPARATOR=\n
... |--------------|-------------|-------------|
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}name${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}description${SPACE}|${SPACE}subnetworks${SPACE}|
... |--------------|-------------|-------------|
... |${SPACE}robot-vpc-01${SPACE}|${SPACE}null${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}null${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
... |--------------|-------------|-------------|
Stock Stackql Exec Inline Equals Both Streams
... ${getNetworkDetailQueryStr}
... ${networkZeroSubnetResult}
... ${EMPTY}
... Google-Network-Lifecycle-Select-Network-02
Sleep 60s
Stock Stackql Exec Inline Equals Both Streams
... ${deleteNetworkStr}
... ${EMPTY}
... ${EMPTY}
... Google-Network-Lifecycle-Network-Delete
Sleep 20s
${networkEmptyResult} = Catenate SEPARATOR=\n
... |------|-------------|-------------|
... |${SPACE}name${SPACE}|${SPACE}description${SPACE}|${SPACE}subnetworks${SPACE}|
... |------|-------------|-------------|
Stock Stackql Exec Inline Equals Both Streams
... ${getNetworkDetailQueryStr}
... ${networkEmptyResult}
... ${EMPTY}
... Google-Network-Lifecycle-Select-Network-03


Loading
Loading