net-istio flaky conformance tests

[AngeloDanducci]

Failing tests can be found here

TestIngressConformance/Update/# seems to be the primary culprit, with the issue
Reason:"NewObservedGenFailure", Message:"unsuccessfully observed a new generation"

Will dig into this a bit more and update the issue with any further findings.

[AngeloDanducci]

/assign @AngeloDanducci

[AngeloDanducci]

Looks like it may be related to this as the observed generation isn't incrementing.
istio/istio#31931

[AngeloDanducci]

/lifecycle frozen

[dprotaso]

NewObservedGenFailure is actually an error that the knative controllers set

https://github.com/knative/pkg/blob/d1d5c849073b4c59fabeef1aaaf5eece021d9ee2/reconciler/reconcile_common.go#L68-L72

net-istio should rebuild the status conditions - but I wonder if something is causing them to short circuit reconciliation and not do that

[dprotaso]

We don't use Istio's Virtual Service Status as source of information since it was always broken - https://github.com/knative-sandbox/net-istio/blob/abad502be7e14ef7281de0d77dcf5525d1e7e40b/config/400-config-istio.yaml#L67-L70

So I think the flake is specific to our controllers not re-creating the KIngress Status block in some situation.

It still remains flakey in testgrid