After upgrading controller-runtime version, the test function using fake client paniced

[zoroyihan]

After upgrading controller-runtime version from v0.21.0 to v0.22.4.
A previously working test that used fakeclient is now panicking.

panic log:

panic: reflect: call of reflect.Value.Field on ptr Value [recovered]
panic: reflect: call of reflect.Value.Field on ptr Value

goroutine 89 [running]:
testing.tRunner.func1.2({0x47a9be0, 0xc001197de8})
/usr/local/go/src/testing/testing.go:1734 +0x22a
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1737 +0x428
panic({0x47a9be0?, 0xc001197de8?})
/usr/local/go/src/runtime/panic.go:792 +0x136
reflect.Value.Field({0x4838240?, 0xc000dda170?, 0x4838240?}, 0x0)
/usr/local/go/src/reflect/value.go:1272 +0x197
reflect.Value.FieldByIndex({0x4838240?, 0xc000dda170?, 0x45e3c00?}, {0x51c7d20?, 0x2199b65?, 0x45e3c00?})
/usr/local/go/src/reflect/value.go:1305 +0x71
sigs.k8s.io/structured-merge-diff/v6/value.(*FieldCacheEntry).GetFrom(0x0?, {0x4cd4980?, 0xc000dda110?, 0x14?})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/reflectcache.go:87 +0x6c
sigs.k8s.io/structured-merge-diff/v6/value.eachStructField({0x4cd4980?, 0xc000dda110?, 0xc0010e5a58?}, 0xc0010e5a40)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/structreflect.go:118 +0xeb
sigs.k8s.io/structured-merge-diff/v6/value.structReflect.IterateUsing({{0x0, 0x0, {0x4cd4980, 0xc000dda110, 0x199}, 0x1}}, {0x5172610, 0xc001b90a40}, 0xc001b8dee0)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/structreflect.go:111 +0xfc
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).visitMapItems(0xc000a91140, 0xc000513700, {0x5186d80, 0xc000a852c0})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:174 +0xaf
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).doMap(0xc000a91140, 0xc000513700)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:202 +0x125
sigs.k8s.io/structured-merge-diff/v6/typed.handleAtom({0x0?, 0x0?, 0xc000513700?}, {0xc0005196d0, {0x0, 0x0, 0x0}, 0x0}, {0x514cc40, 0xc000a91140})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/helpers.go:145 +0x10f
sigs.k8s.io/structured-merge-diff/v6/typed.resolveSchema(0x0?, {0xc0005196d0, {0x0, 0x0, 0x0}, 0x0}, {0x518ae80, 0xc000a85290}, {0x514cc40, 0xc000a91140})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/helpers.go:116 +0x119
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).validate(0xc000a910e0?, 0xc0010e5e48)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:85 +0x6d
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).visitMapItems.func1({0x445b535, 0x4}, {0x518ae80, 0xc000a85290})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:186 +0x23b
sigs.k8s.io/structured-merge-diff/v6/value.structReflect.IterateUsing.func1(0xc000ad94f0?, {0x445b535, 0x4}, {0x4cd4980?, 0xc000dda110?, 0x7fb162f3f048?})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/structreflect.go:112 +0x64
sigs.k8s.io/structured-merge-diff/v6/value.eachStructField({0x4ae5080?, 0xc000dda008?, 0xc0010e5ff8?}, 0xc0010e5fe0)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/structreflect.go:123 +0x13e
sigs.k8s.io/structured-merge-diff/v6/value.structReflect.IterateUsing({{0x0, 0x0, {0x4ae5080, 0xc000dda008, 0x199}, 0x1}}, {0x5172610, 0xc001b90a40}, 0xc001b8dd60)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/value/structreflect.go:111 +0xfc
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).visitMapItems(0xc000a910e0, 0xc000513700, {0x5186d80, 0xc000a85260})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:174 +0xaf
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).doMap(0xc000a910e0, 0xc000513700)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:202 +0x125
sigs.k8s.io/structured-merge-diff/v6/typed.handleAtom({0x0?, 0x0?, 0xc000513700?}, {0xc000519710, {0x0, 0x0, 0x0}, 0x0}, {0x514cc40, 0xc000a910e0})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/helpers.go:145 +0x10f
sigs.k8s.io/structured-merge-diff/v6/typed.resolveSchema(0x1e440a5?, {0xc000519710, {0x0, 0x0, 0x0}, 0x0}, {0x518ae80, 0xc000da3410}, {0x514cc40, 0xc000a910e0})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/helpers.go:116 +0x119
sigs.k8s.io/structured-merge-diff/v6/typed.(*validatingObjectWalker).validate(0x518ae80?, 0x0)
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/validate.go:85 +0x6d
sigs.k8s.io/structured-merge-diff/v6/typed.TypedValue.Validate({{0x518ae80, 0xc000da3410}, {0xc000519710, {0x0, 0x0, 0x0}, 0x0}, 0xc000381340}, {0xc000dd83b0, 0x1, ...})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/typed.go:114 +0xfb
sigs.k8s.io/structured-merge-diff/v6/typed.AsTyped({0x518ae80, 0xc000da3410}, 0xc000381340, {0xc000519710, {0x0, 0x0, 0x0}, 0x0}, {0xc000dd83b0, 0x1, ...})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/typed.go:59 +0x185
sigs.k8s.io/structured-merge-diff/v6/typed.ParseableType.FromStructured({{0xc000519710, {0x0, 0x0, 0x0}, 0x0}, 0xc000381340}, {0x4e27280?, 0xc000dda008?}, {0xc000dd83b0, 0x1, ...})
${GOPATH}/pkg/mod/sigs.k8s.io/structured-merge-diff/v6@v6.3.0/typed/parser.go:125 +0xff
k8s.io/apimachinery/pkg/util/managedfields/internal.deducedTypeConverter.ObjectToTyped({}, {0x5134478?, 0xc000dda008?}, {0xc000dd83b0?, 0x4?, 0xc00134fb20?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/typeconverter.go:92 +0x12e
sigs.k8s.io/controller-runtime/pkg/client/fake.multiTypeConverter.ObjectToTyped({{0xc00173d160?, 0xc0010e65f0?, 0x2e1cc79?}}, {0x5134478, 0xc000dda008}, {0xc000dd83b0, 0x1, 0x1})
${GOPATH}/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/client/fake/typeconverter.go:35 +0xde
k8s.io/apimachinery/pkg/util/managedfields/internal.(*structuredMergeManager).Update(0xc000a8cfa0, {0x5134478, 0xc000d9bb08}, {0x5134478, 0xc000d9b208}, {0x7fb162f83078, 0xc00134fb10}, {0x4e72020, 0x11})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/structuredmerge.go:100 +0x284
k8s.io/apimachinery/pkg/util/managedfields/internal.(*stripMetaManager).Update(0xc001197830, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0x4e72020, 0x11})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/stripmeta.go:58 +0x64
k8s.io/apimachinery/pkg/util/managedfields/internal.(*managedFieldsUpdater).Update(0xc001b91340?, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0xc000718d20, 0x55})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/managedfieldsupdater.go:47 +0x74
k8s.io/apimachinery/pkg/util/managedfields/internal.(*buildManagerInfoManager).Update(0xc001b91340, {0x5134478, 0xc000d9bb08}, {0x5134478, 0xc000d9b208}, {0x7fb162f83078, 0xc00134fb10}, {0x0?, 0x1f0041f?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/buildmanagerinfo.go:51 +0xfb
k8s.io/apimachinery/pkg/util/managedfields/internal.(*capManagersManager).Update(0xc000da32c0, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0x0?, 0x4000?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/capmanagers.go:48 +0x5c
k8s.io/apimachinery/pkg/util/managedfields/internal.(*skipNonAppliedManager).Update(0xc001b91380, {0x5134478, 0xc000d9bb08}, {0x5134478, 0xc000d9b208}, {0x7fb162f83078, 0xc00134fb10}, {0x0, 0x0})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/skipnonapplied.go:72 +0x129
k8s.io/apimachinery/pkg/util/managedfields/internal.(*lastAppliedManager).Update(0x2?, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0x0?, 0x4e27280?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/lastappliedmanager.go:53 +0x47
k8s.io/apimachinery/pkg/util/managedfields/internal.(*lastAppliedUpdater).Update(0xc0010e6aa8?, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0x0?, 0xc00134fb10?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/lastappliedupdater.go:43 +0x47
k8s.io/apimachinery/pkg/util/managedfields/internal.(*versionCheckManager).Update(0x5134478?, {0x5134478?, 0xc000d9bb08?}, {0x5134478?, 0xc000d9b208?}, {0x7fb162f83078?, 0xc00134fb10?}, {0x0?, 0x2e91920?})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/versioncheck.go:43 +0x47
k8s.io/apimachinery/pkg/util/managedfields/internal.(*FieldManager).Update(0xc001b8dce0, {0x5134478, 0xc000d9bb08}, {0x5134478, 0xc000d9b208}, {0x0, 0x0})
${GOPATH}/pkg/mod/k8s.io/apimachinery@v0.34.1/pkg/util/managedfields/internal/fieldmanager.go:131 +0xdf
k8s.io/client-go/testing.(*managedFieldObjectTracker).Patch(0xc000a4c870, {{0x4e86d0f, 0x18}, {0x4e444e2, 0x2}, {0xc000a897f8, 0x5}}, {0x5134478, 0xc000d9b208}, {0x0, ...}, ...)
${GOPATH}/pkg/mod/k8s.io/client-go@v0.34.1/testing/fixture.go:751 +0x345
sigs.k8s.io/controller-runtime/pkg/client/fake.versionedTracker.Patch({{0x517a3c0, 0xc000a4c870}, 0xc00037b7a0, 0xc0009feed0, 0x1}, {{0x4e86d0f, 0x18}, {0x4e444e2, 0x2}, {0xc000a897f8, ...}}, ...)
${GOPATH}/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/client/fake/client.go:515 +0x31c
k8s.io/client-go/testing.objectTrackerReact.Patch({{_, _}}, {{{0x0, 0x0}, {0x4e47a39, 0x5}, {{0x4e86d0f, 0x18}, {0x4e444e2, 0x2}, ...}, ...}, ...})
${GOPATH}/pkg/mod/k8s.io/client-go@v0.34.1/testing/fixture.go:280 +0x739
k8s.io/client-go/testing.ObjectReaction.func1({0x5169428?, 0xc000217200?})
${GOPATH}/pkg/mod/k8s.io/client-go@v0.34.1/testing/fixture.go:128 +0x408
sigs.k8s.io/controller-runtime/pkg/client/fake.(*fakeClient).patch(0xc0008855f0, {0x51a9d78, 0xc000d9a608}, {0x51341f8, 0xc000d83bc0}, {0x0, 0x0, 0x0})
${GOPATH}/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/client/fake/client.go:1228 +0xa07
sigs.k8s.io/controller-runtime/pkg/client/fake.(*fakeClient).Patch(0x149?, {0x160?, 0x100?}, {0x51a9d78?, 0xc000d9a608?}, {0x51341f8?, 0xc000d83bc0?}, {0x0, 0x0, 0x0})
${GOPATH}/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.4/pkg/client/fake/client.go:1071 +0x4e
... my own code

The fake client is initialized like this:
crd := &v1.MyCRD{ObjectMeta: metav1.ObjectMeta{Name: name}, ...} fake.NewClientBuilder().WithScheme(scheme.Scheme).WithObjects(crd).WithStatusSubresource(crd).Build()

The place where the call is failing is written like this:
err = r.Patch(ctx, &v1.MyCRD{ObjectMeta: metav1.ObjectMeta{Name: name}}, client.RawPatch(types.MergePatchType, patchData))

Can someone help to answer? thks

[alvaroaleman]

@zoroyihan this is too little information, please provide a full reproducer that someone else can run (a testcase is fine), otherwise we won't be able to debug this.

[soorya38]

Hi @maintainer,
I’m interested in picking this up and contributing a fix.
If it’s available, I’d appreciate being assigned to this issue.

Thank you!

[alvaroaleman]

@soorya38 there is nothing to pick up at the moment, the information the author has provided is insufficient to reproduce the issue.

[AndrewChubatiuk]

@alvaroaleman @soorya38
i have this issue, in this commit I've replaced structured-merge-diff dependency with code from fork to fixes it. to reproduce just need to remove that replacement from go.mod and run go test ./internal/controller/operator/factory/vmdistributed/... from repo root

[alvaroaleman]

Okay, we can not change the structured merge diff version, it comes in transiently through client-go so this must be fixed in client-go. If this worked previously, can you open an issue in kubernetes/kubernetes and mark it as a regression?

[AndrewChubatiuk]

In my case this is not a regression. Tried version v0.21.0, but it's still there

[alvaroaleman]

In my case this is not a regression. Tried version v0.21.0, but it's still there

the FieldManagedObjectTracker which is what uses structured-merge-diff in the fake client wasn't used by the fake client in 0.21, are you sure?

[alvaroaleman]

i have this issue, in this commit I've replaced structured-merge-diff dependency with code from fork to fixes it. to reproduce just need to remove that replacement from go.mod and run go test ./internal/controller/operator/factory/vmdistributed/... from repo root

Could you extract a minimal repro from that and check if what you are trying to do work with a real Kubernetes apiserver? The code that is used for fieldmanagement is the same between the tracker and the kube-apiserver.

[AndrewChubatiuk]

sorry, you're right, it's working with 0.21.0

[alvaroaleman]

A trivial reproducer is something like this:

$ g diff pkg/client/fake/client_test.go
diff --git a/pkg/client/fake/client_test.go b/pkg/client/fake/client_test.go
index 209ccc67..2cdbf1ec 100644
--- a/pkg/client/fake/client_test.go
+++ b/pkg/client/fake/client_test.go
@@ -22,8 +22,10 @@ import (
        "fmt"
        "strconv"
        "sync"
+       stdlibtesting "testing"
        "time"

+       vmv1beta1 "github.com/VictoriaMetrics/operator/api/operator/v1beta1"
        "github.com/google/go-cmp/cmp"
        . "github.com/onsi/ginkgo/v2"
        . "github.com/onsi/gomega"
@@ -3371,3 +3373,15 @@ var _ = Describe("Fake client builder", func() {
                }).To(Panic())
        })
 })
+
+func TestPanic(t *stdlibtesting.T) {
+       if err := vmv1beta1.AddToScheme(clientgoscheme.Scheme); err != nil {
+               t.Fatalf("Failed to add vmv1beta1 to scheme: %v", err)
+       }
+       agent := &vmv1beta1.VMAgent{
+               ObjectMeta: metav1.ObjectMeta{Name: "foobar"},
+       }
+       c := NewClientBuilder().WithObjects(agent).WithStatusSubresource(agent).Build()
+       agent.Labels = map[string]string{"foo": "bar"}
+       c.Patch(t.Context(), agent, client.MergeFrom(agent))
+}

So it seems like something in this CRD trips the ManagedFieldObjectTracker/structured-merge-diff, it would be good to understand what exactly