Skip to content

✨ Add basic support for mutating object contents during syncing #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 7, 2025
Merged

✨ Add basic support for mutating object contents during syncing #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e29acd5
bring back support for mutations, remove Rudi support
xrstf Feb 6, 2025
40aa688
update docs
xrstf Feb 6, 2025
4bef5f3
codegen
xrstf Feb 6, 2025
5d7e49e
go mod tidy
xrstf Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 156 additions & 0 deletions deploy/crd/kcp.io/syncagent.kcp.io_publishedresources.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,84 @@ spec:
type: object
x-kubernetes-map-type: atomic
type: object
mutation:
description: |-
Mutation allows to configure "rewrite rules" to modify the objects in both
directions during the synchronization.
properties:
spec:
items:
properties:
delete:
properties:
path:
type: string
required:
- path
type: object
regex:
properties:
path:
type: string
pattern:
description: |-
Pattern can be left empty to simply replace the entire value with the
replacement.
type: string
replacement:
type: string
required:
- path
type: object
template:
properties:
path:
type: string
template:
type: string
required:
- path
- template
type: object
type: object
type: array
status:
items:
properties:
delete:
properties:
path:
type: string
required:
- path
type: object
regex:
properties:
path:
type: string
pattern:
description: |-
Pattern can be left empty to simply replace the entire value with the
replacement.
type: string
replacement:
type: string
required:
- path
type: object
template:
properties:
path:
type: string
template:
type: string
required:
- path
- template
type: object
type: object
type: array
type: object
naming:
description: |-
Naming can be used to control how the namespace and names for local objects
Expand Down Expand Up @@ -251,6 +329,84 @@ spec:
kind:
description: ConfigMap or Secret
type: string
mutation:
description: |-
Mutation configures optional transformation rules for the related resource.
Status mutations are only performed when the related resource originates in kcp.
properties:
spec:
items:
properties:
delete:
properties:
path:
type: string
required:
- path
type: object
regex:
properties:
path:
type: string
pattern:
description: |-
Pattern can be left empty to simply replace the entire value with the
replacement.
type: string
replacement:
type: string
required:
- path
type: object
template:
properties:
path:
type: string
template:
type: string
required:
- path
- template
type: object
type: object
type: array
status:
items:
properties:
delete:
properties:
path:
type: string
required:
- path
type: object
regex:
properties:
path:
type: string
pattern:
description: |-
Pattern can be left empty to simply replace the entire value with the
replacement.
type: string
replacement:
type: string
required:
- path
type: object
template:
properties:
path:
type: string
template:
type: string
required:
- path
- template
type: object
type: object
type: array
type: object
origin:
description: '"service" or "kcp"'
type: string
Expand Down
15 changes: 0 additions & 15 deletions docs/publish-resources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,6 @@ spec:
name: "cert-$remoteNamespaceHash-$remoteNameHash"
```

<!--
### Mutation

Besides projecting the type meta, changes to object contents are also nearly always required.
Expand Down Expand Up @@ -182,7 +181,6 @@ spec:
- regex: ...
template: ...
delete: ...
rudi: ...
```

#### Regex
Expand Down Expand Up @@ -218,19 +216,6 @@ delete:
This mutation simply removes the value at the given path from the document. JSON path is the
usual path, without a leading dot.

#### Rudi

```yaml
rudi:
script: |
(set! .metadata.name (concat "-" [$localObj.metadata.name "foo"]))
```

This mutation runs a [Rudi](https://github.com/xrstf/rudi) program on the document. Note that in
Rudi, in contrast to all the other mutation types, JSON paths _do_ begin with leading dot if you
refer to the main document. Just `metadata.name` would be invalid.
-->

### Related Resources

The processing of resources on the service cluster often leads to additional resources being
Expand Down
1 change: 0 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ require (
github.com/tidwall/gjson v1.18.0
github.com/tidwall/sjson v1.2.5
go.uber.org/zap v1.27.0
go.xrstf.de/rudi v0.0.9
Copy link
Member

@embik embik Feb 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😢

k8c.io/reconciler v0.5.0
k8s.io/api v0.31.2
k8s.io/apiextensions-apiserver v0.31.2
Expand Down
2 changes: 0 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,8 +359,6 @@ go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN8
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
go.xrstf.de/rudi v0.0.9 h1:ACTYQJcq8Kk24yPGDhApMVwGH5JrgBLQdvkrwJXUt3c=
go.xrstf.de/rudi v0.0.9/go.mod h1:ERo0X1RhWc5J8FFlNWx9i0j3ZEvrRD/YXqVvo+q1rfo=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
Expand Down
2 changes: 1 addition & 1 deletion internal/controller/sync/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ func Create(
}

// create the syncer that holds the meat&potatoes of the synchronization logic
mutator := mutation.NewMutator(nil) // pubRes.Spec.Mutation
mutator := mutation.NewMutator(pubRes.Spec.Mutation)
syncer, err := sync.NewResourceSyncer(log, localManager.GetClient(), virtualWorkspaceCluster.GetClient(), pubRes, localCRD, apiExportName, mutator, stateNamespace, agentName)
if err != nil {
return nil, fmt.Errorf("failed to create syncer: %w", err)
Expand Down
31 changes: 1 addition & 30 deletions internal/mutation/mutation.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ import (
"github.com/Masterminds/sprig/v3"
"github.com/tidwall/gjson"
"github.com/tidwall/sjson"
"go.xrstf.de/rudi"

syncagentv1alpha1 "github.com/kcp-dev/api-syncagent/sdk/apis/syncagent/v1alpha1"
)
Expand All @@ -46,11 +45,6 @@ func ApplyResourceMutations(value any, mutations []syncagentv1alpha1.ResourceMut
}

func ApplyResourceMutation(value any, mut syncagentv1alpha1.ResourceMutation, ctx *TemplateMutationContext) (any, error) {
// for Rudi scripts we can skip all the JSON encoding/decoding
if mut.Rudi != nil {
return applyResourceRudiMigration(value, *mut.Rudi, ctx)
}

// encode current value as JSON
encoded, err := json.Marshal(value)
if err != nil {
Expand Down Expand Up @@ -82,33 +76,10 @@ func applyResourceMutationToJSON(jsonData string, mut syncagentv1alpha1.Resource
case mut.Regex != nil:
return applyResourceRegexMutation(jsonData, *mut.Regex)
default:
return "", errors.New("must use either Rudi, regex, template or delete mutation")
return "", errors.New("must use either regex, template or delete mutation")
}
}

func applyResourceRudiMigration(value any, mut syncagentv1alpha1.ResourceRudiMutation, ctx *TemplateMutationContext) (any, error) {
program, err := rudi.Parse("myscript", mut.Script)
if err != nil {
return nil, fmt.Errorf("invalid script: %w", err)
}

funcs := rudi.NewBuiltInFunctions()
vars := rudi.NewVariables()

if ctx != nil {
vars.
Set("localObj", ctx.LocalObject).
Set("remoteObj", ctx.RemoteObject)
}

processed, _, err := program.Run(value, vars, funcs)
if err != nil {
return nil, fmt.Errorf("script failed: %w", err)
}

return processed, nil
}

func applyResourceDeleteMutation(jsonData string, mut syncagentv1alpha1.ResourceDeleteMutation) (string, error) {
jsonData, err := sjson.Delete(jsonData, mut.Path)
if err != nil {
Expand Down
83 changes: 0 additions & 83 deletions internal/mutation/mutation_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,89 +187,6 @@ func TestApplyResourceMutation(t *testing.T) {
},
expected: `{"spec":[1,3]}`,
},

// Rudi

{
name: "Rudi: empty script",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `.`,
},
},
expected: `{"spec":{"secretName":"foo"}}`,
},
{
name: "Rudi: set one new key",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(set! .foo "bar")`,
},
},
expected: `{"foo":"bar","spec":{"secretName":"foo"}}`,
},
{
name: "Rudi: update existing key",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(set! .spec.secretName "bar")`,
},
},
expected: `{"spec":{"secretName":"bar"}}`,
},
{
name: "Rudi: remove a key",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(delete! .spec.secretName)`,
},
},
expected: `{"spec":{}}`,
},
{
name: "Rudi: result value is ignored, only document counts",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(delete! .spec.secretName) false`,
},
},
expected: `{"spec":{}}`,
},
{
name: "Rudi: local object becomes $localObj",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(set! .spec $localObj.local)`,
},
},
ctx: &TemplateMutationContext{
LocalObject: map[string]any{
"local": "data",
},
},
expected: `{"spec":"data"}`,
},
{
name: "Rudi: remote object becomes $remoteObj",
inputData: `{"spec":{"secretName":"foo"}}`,
mutation: syncagentv1alpha1.ResourceMutation{
Rudi: &syncagentv1alpha1.ResourceRudiMutation{
Script: `(set! .spec $remoteObj.remote)`,
},
},
ctx: &TemplateMutationContext{
RemoteObject: map[string]any{
"remote": "data",
},
},
expected: `{"spec":"data"}`,
},
}

for _, testcase := range testcases {
Expand Down
2 changes: 1 addition & 1 deletion internal/sync/syncer_related.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ func (s *ResourceSyncer) processRelatedResource(log *zap.SugaredLogger, stateSto
// sure we can clean up properly
blockSourceDeletion: relRes.Origin == "kcp",
// apply mutation rules configured for the related resource
mutator: mutation.NewMutator(nil), // relRes.Mutation
mutator: mutation.NewMutator(relRes.Mutation),
}

requeue, err = syncer.Sync(log, sourceSide, destSide)
Expand Down
Loading