Skip to content

Commit 414a233

Browse files
derek10cloudgeneral-kroll-4-life
derek10cloud
and
general-kroll-4-life
authored
support overwrite with registry pull (#550)
* support overwrite with registry pull * fix: Format error * skip all of the breakable build and push stuff in fork mode * fix formatting error * robot-test-coverage-registry-overwrite Summary: - Changed existing mocked registry entries for point of difference, in otder to support meaningful testing. - Force drop registry cache entry for a given provider on `registry pull`. - Added robot test `Registry Pull Overwrite Works in Both Directions`. * update-any-sdk Summary: - Update `anny-sdk` version. - Attempt support for windows provider delete. * update-any-sdk Summary: - Update `anny-sdk` version. - Fix file locking issue with signature check. - Attempt support for windows provider delete. * update-any-sdk Summary: - Update `anny-sdk` version. - Eager close on provider file itself, inside `any-sdk`. * update-any-sdk Summary: - Update `anny-sdk` version. - Eager close on provider file itself, inside `any-sdk`. --------- Co-authored-by: General Kroll <[email protected]> Co-authored-by: Benevolent General Kroll Who cannot spell <[email protected]>
1 parent 6e37779 commit 414a233

File tree

10 files changed

+148
-29
lines changed

10 files changed

+148
-29
lines changed

‎.github/workflows/build.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1537,6 +1537,7 @@ jobs:
15371537
name: digests-${{ env.PLATFORM_PAIR }}
15381538

15391539
- name: Prepare digest information
1540+
if: env.BUILD_IMAGE_REQUIRED == 'true'
15401541
working-directory: ${{ runner.temp }}/digests
15411542
run: |
15421543
platform=${{ matrix.platform }}

‎.vscode/launch.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,7 @@
177177
"delete from aws.cloud_control.resources where region = 'ap-southeast-1' and data__TypeName = 'AWS::Logs::LogGroup' and data__Identifier = 'LogGroupResourceExampleThird' ;",
178178
"insert into google.storage.buckets( project, data__name) select 'testing-project', 'silly-bucket' returning projectNumber;",
179179
"insert /*+ AWAIT */ into google.compute.networks(project, data__name, data__autoCreateSubnetworks) select 'mutable-project', 'auto-test-01', false returning creationTimestamp, name;",
180+
"registry pull google 'v0.1.2'; show resources in google.storage; registry pull google 'v0.1.1-alpha01'; show resources in google.storage; registry pull google 'v0.1.0'; show resources in google.storage;",
180181
],
181182
"default": "show providers;"
182183
},
@@ -195,7 +196,8 @@
195196
"{ \"url\": \"https://cdn.statically.io/gh/stackql/stackql-provider-registry/main/providers\", \"localDocRoot\": \"${workspaceFolder}/test/registry\" }",
196197
"{ \"url\": \"https://cdn.statically.io/gh/stackql/stackql-provider-registry/dev/providers\" }",
197198
"{ \"url\": \"https://registry-dev.stackql.app/providers\" }",
198-
"{ \"url\": \"https://registry.stackql.app/providers\" }"
199+
"{ \"url\": \"https://registry.stackql.app/providers\" }",
200+
"{\"url\": \"http://localhost:1094/gh/stackql/stackql-provider-registry/main/providers\", \"verifyConfig\": {\"nopVerify\": true}}",
199201
],
200202
"default": "{ \"url\": \"file://${workspaceFolder}/test/registry\", \"localDocRoot\": \"${workspaceFolder}/test/registry\", \"verifyConfig\": { \"nopVerify\": true } }"
201203
},

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ require (
1919
github.com/spf13/cobra v1.4.0
2020
github.com/spf13/pflag v1.0.5
2121
github.com/spf13/viper v1.10.1
22-
github.com/stackql/any-sdk v0.1.4-alpha06
22+
github.com/stackql/any-sdk v0.1.4-alpha11
2323
github.com/stackql/go-suffix-map v0.0.1-alpha01
2424
github.com/stackql/psql-wire v0.1.1-beta23
2525
github.com/stackql/stackql-parser v0.0.15-alpha06

‎go.sum‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -484,8 +484,8 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
484484
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
485485
github.com/spf13/viper v1.10.1 h1:nuJZuYpG7gTj/XqiUwg8bA0cp1+M2mC3J4g5luUYBKk=
486486
github.com/spf13/viper v1.10.1/go.mod h1:IGlFPqhNAPKRxohIzWpI5QEy4kuI7tcl5WvR+8qy1rU=
487-
github.com/stackql/any-sdk v0.1.4-alpha06 h1:QJPf3ehPrRqmYZR+TmD897AsmsaOamHErLhaE5B9v/w=
488-
github.com/stackql/any-sdk v0.1.4-alpha06/go.mod h1:AKS/g28y7m4SWL/YW8veE9MCNy8XJgaicVibemVE9e8=
487+
github.com/stackql/any-sdk v0.1.4-alpha11 h1:WeFvLcml37sq+HbVk3hiXuImQ578SE9LM/K1MzqFovo=
488+
github.com/stackql/any-sdk v0.1.4-alpha11/go.mod h1:AKS/g28y7m4SWL/YW8veE9MCNy8XJgaicVibemVE9e8=
489489
github.com/stackql/go-suffix-map v0.0.1-alpha01 h1:TDUDS8bySu41Oo9p0eniUeCm43mnRM6zFEd6j6VUaz8=
490490
github.com/stackql/go-suffix-map v0.0.1-alpha01/go.mod h1:QAi+SKukOyf4dBtWy8UMy+hsXXV+yyEE4vmBkji2V7g=
491491
github.com/stackql/psql-wire v0.1.1-beta23 h1:1ayYMjZArfDcIMyEOKnm+Bp1zRCISw8pguvTFuUhhVQ=

‎internal/stackql/handler/handler.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ type HandlerContext interface { //nolint:revive // don't mind stuttering this on
4848
GetAuthContext(providerName string) (*dto.AuthCtx, error)
4949
GetDBMSInternalRouter() dbmsinternal.Router
5050
GetProvider(providerName string) (provider.IProvider, error)
51+
DeleteProvider(providerName string) error
5152
GetSupportedProviders(extended bool) (map[string]map[string]interface{}, error)
5253
LogHTTPResponseMap(target interface{})
5354
//
@@ -286,6 +287,23 @@ func (hc *standardHandlerContext) GetASTFormatter() sqlparser.NodeFormatter {
286287
return hc.formatter
287288
}
288289

290+
func (hc *standardHandlerContext) DeleteProvider(providerName string) error {
291+
hc.providersMapMutex.Lock()
292+
defer hc.providersMapMutex.Unlock()
293+
if providerName == "googleapis.com" {
294+
providerName = "google"
295+
}
296+
if providerName == "" {
297+
return fmt.Errorf("provider name cannot be empty")
298+
}
299+
_, ok := hc.providers[providerName]
300+
if len(hc.providers) == 0 || !ok {
301+
return fmt.Errorf("provider '%s' not found", providerName)
302+
}
303+
delete(hc.providers, providerName)
304+
return nil
305+
}
306+
289307
func (hc *standardHandlerContext) GetProvider(providerName string) (provider.IProvider, error) {
290308
var err error
291309
if providerName == "" {

‎internal/stackql/planbuilder/plan_builder.go‎

Lines changed: 52 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -647,9 +647,7 @@ func (pgb *standardPlanGraphBuilder) handleDelete(pbi planbuilderinput.PlanBuild
647647
}
648648

649649
//nolint:gocognit // acceptable
650-
func (pgb *standardPlanGraphBuilder) handleRegistry(
651-
pbi planbuilderinput.PlanBuilderInput,
652-
) error {
650+
func (pgb *standardPlanGraphBuilder) handleRegistry(pbi planbuilderinput.PlanBuilderInput) error {
653651
handlerCtx := pbi.GetHandlerCtx()
654652
node, ok := pbi.GetRegistry()
655653
if !ok {
@@ -665,29 +663,10 @@ func (pgb *standardPlanGraphBuilder) handleRegistry(
665663
return err
666664
}
667665
pr := primitive.NewLocalPrimitive(
668-
//nolint:revive // acceptable for now
669-
func(pc primitive.IPrimitiveCtx) internaldto.ExecutorOutput {
666+
func(_ primitive.IPrimitiveCtx) internaldto.ExecutorOutput {
670667
switch at := strings.ToLower(node.ActionType); at {
671668
case "pull":
672-
providerVersion := node.ProviderVersion
673-
if providerVersion == "" {
674-
providerVersion, err = reg.GetLatestPublishedVersion(node.ProviderId)
675-
}
676-
if err != nil {
677-
return internaldto.NewErroneousExecutorOutput(err)
678-
}
679-
err = reg.PullAndPersistProviderArchive(node.ProviderId, providerVersion)
680-
if err != nil {
681-
return internaldto.NewErroneousExecutorOutput(err)
682-
}
683-
return util.PrepareResultSet(
684-
internaldto.NewPrepareResultSetPlusRawDTO(
685-
nil, nil, nil, nil, nil,
686-
internaldto.NewBackendMessages([]string{fmt.Sprintf(
687-
"%s provider, version '%s' successfully installed",
688-
node.ProviderId, providerVersion)}),
689-
nil,
690-
pbi.GetHandlerCtx().GetTypingConfig()))
669+
return pgb.handleRegistryPull(reg, node, pbi)
691670
case "list":
692671
var colz []string
693672
var provz map[string]anysdk.ProviderDescription
@@ -737,10 +716,58 @@ func (pgb *standardPlanGraphBuilder) handleRegistry(
737716
},
738717
)
739718
pgb.planGraphHolder.CreatePrimitiveNode(pr)
740-
741719
return nil
742720
}
743721

722+
// handleRegistryPull is a helper function that pulls a provider version from the registry and persists it locally.
723+
func (pgb *standardPlanGraphBuilder) handleRegistryPull(
724+
reg anysdk.RegistryAPI,
725+
node *sqlparser.Registry,
726+
pbi planbuilderinput.PlanBuilderInput,
727+
) internaldto.ExecutorOutput {
728+
providerVersion := node.ProviderVersion
729+
var err error
730+
if providerVersion == "" {
731+
providerVersion, err = reg.GetLatestPublishedVersion(node.ProviderId)
732+
if err != nil {
733+
return internaldto.NewErroneousExecutorOutput(err)
734+
}
735+
}
736+
// Get all existing versions from local filesystem
737+
localProviders := reg.ListLocallyAvailableProviders()
738+
// Handle special case for google provider
739+
providerID := node.ProviderId
740+
if providerID == "google" {
741+
providerID = "googleapis.com"
742+
}
743+
if providerDesc, exists := localProviders[providerID]; exists {
744+
// Remove all existing versions
745+
for _, version := range providerDesc.Versions {
746+
if err = reg.RemoveProviderVersion(providerID, version); err != nil {
747+
return internaldto.NewErroneousExecutorOutput(err)
748+
}
749+
}
750+
}
751+
// Clear provider cache before pulling new version
752+
if err = reg.ClearProviderCache(providerID); err != nil {
753+
return internaldto.NewErroneousExecutorOutput(err)
754+
}
755+
handlerCtx := pbi.GetHandlerCtx()
756+
_ = handlerCtx.DeleteProvider(providerID)
757+
// Pull and persist the requested version
758+
if err = reg.PullAndPersistProviderArchive(node.ProviderId, providerVersion); err != nil {
759+
return internaldto.NewErroneousExecutorOutput(err)
760+
}
761+
return util.PrepareResultSet(
762+
internaldto.NewPrepareResultSetPlusRawDTO(
763+
nil, nil, nil, nil, nil,
764+
internaldto.NewBackendMessages([]string{fmt.Sprintf(
765+
"%s provider, version '%s' successfully installed",
766+
node.ProviderId, providerVersion)}),
767+
nil,
768+
pbi.GetHandlerCtx().GetTypingConfig()))
769+
}
770+
744771
func (pgb *standardPlanGraphBuilder) handlePurge(pbi planbuilderinput.PlanBuilderInput) error {
745772
handlerCtx := pbi.GetHandlerCtx()
746773
node, ok := pbi.GetPurge()

‎test/python/stackql_test_tooling/flask/registry/expectations.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,18 @@
3737
"status": 200,
3838
"file": "google-v0.1.1-alpha01.tgz"
3939
}
40+
},
41+
{
42+
"httpRequest": {
43+
"method": "GET",
44+
"path": "/gh/stackql/stackql-provider-registry/main/providers/dist/googleapis.com/v0.1.0.tgz"
45+
},
46+
"httpResponse": {
47+
"headers": {
48+
"Content-Type": "application/gzip"
49+
},
50+
"status": 200,
51+
"file": "google-v0.1.0.tgz"
52+
}
4053
}
4154
]

‎test/python/stackql_test_tooling/flask/registry/templates/google-v0.1.0.tgz‎

1.67 KB
Binary file not shown.

‎test/python/stackql_test_tooling/flask/registry/templates/google-v0.1.1-alpha01.tgz‎

-8.36 KB
Binary file not shown.

‎test/robot/functional/stackql_mocked_from_cmd_line.robot‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2687,6 +2687,64 @@ Registry Pull Google Provider Specific Version Prerelease
26872687
... registry pull google 'v0.1.1\-alpha01' ;
26882688
... successfully installed
26892689

2690+
Registry Pull Overwrite Works in Both Directions
2691+
${inputStr} = Catenate
2692+
... registry pull google 'v0.1.2';
2693+
... show resources in google.storage;
2694+
... registry pull google 'v0.1.1-alpha01';
2695+
... show resources in google.storage;
2696+
... registry pull google 'v0.1.0';
2697+
... show resources in google.storage;
2698+
... registry pull google 'v0.1.1-alpha01';
2699+
... show resources in google.storage;
2700+
... registry pull google 'v0.1.2';
2701+
... show resources in google.storage;
2702+
${stdOutStr} = Catenate SEPARATOR=\n
2703+
... |------|------------------------|
2704+
... |${SPACE}name${SPACE}|${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}id${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
2705+
... |------|------------------------|
2706+
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}google.storage.buckets${SPACE}|
2707+
... |------|------------------------|
2708+
... |------|-------------------------------|
2709+
... |${SPACE}name${SPACE}|${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}id${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
2710+
... |------|-------------------------------|
2711+
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}google.storage.bucket_objects${SPACE}|
2712+
... |------|-------------------------------|
2713+
... |------|--------------------------------|
2714+
... |${SPACE}name${SPACE}|${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}id${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
2715+
... |------|--------------------------------|
2716+
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}google.storage.storage_buckets${SPACE}|
2717+
... |------|--------------------------------|
2718+
... |------|-------------------------------|
2719+
... |${SPACE}name${SPACE}|${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}id${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
2720+
... |------|-------------------------------|
2721+
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}google.storage.bucket_objects${SPACE}|
2722+
... |------|-------------------------------|
2723+
... |------|------------------------|
2724+
... |${SPACE}name${SPACE}|${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}id${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|
2725+
... |------|------------------------|
2726+
... |${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}${SPACE}|${SPACE}google.storage.buckets${SPACE}|
2727+
... |------|------------------------|
2728+
${stdErrStr} = Catenate SEPARATOR=\n
2729+
... google provider, version 'v0.1.2' successfully installed
2730+
... google provider, version 'v0.1.1-alpha01' successfully installed
2731+
... google provider, version 'v0.1.0' successfully installed
2732+
... google provider, version 'v0.1.1-alpha01' successfully installed
2733+
... google provider, version 'v0.1.2' successfully installed
2734+
Should Stackql Exec Inline Equal Both Streams
2735+
... ${STACKQL_EXE}
2736+
... ${OKTA_SECRET_STR}
2737+
... ${GITHUB_SECRET_STR}
2738+
... ${K8S_SECRET_STR}
2739+
... ${REGISTRY_MOCKED_CFG_STR}
2740+
... ${AUTH_CFG_STR}
2741+
... ${SQL_BACKEND_CFG_STR_CANONICAL}
2742+
... ${inputStr}
2743+
... ${stdOutStr}
2744+
... ${stdErrStr}
2745+
... stdout=${CURDIR}/tmp/Registry-Pull-Override-Works-in-Both-Directions.tmp
2746+
... stderr=${CURDIR}/tmp/Registry-Pull-Override-Works-in-Both-Directions-stderr.tmp
2747+
26902748
Registry Pull Google Provider Implicit Latest Version
26912749
Should Stackql Exec Inline Contain Both Streams
26922750
... ${STACKQL_EXE}

0 commit comments

Comments
 (0)