Modifying resources for conformance testing

[nathancoleman]

I'm working on getting the conformance tests running against our (Consul) implementation of the Gateway API. I'm finding some cases where it would be useful to have specific annotations or other things on the resources created for conformance testing. Is this something that anyone else has run into?

I've got a working Deployment patch that adds an annotation using kustomize before running $ make conformance; however, this is obviously somewhat fragile as the testing framework and resources will continue evolving over time. I was curious if anyone else has thought about this and, if so, might it make sense to support this sort of thing officially in some capacity.

This is sort of open-ended, so I thought I'd drop it in Q&A. Looking forward to hearing others' thoughts! 🚀

[robscott]

This is a good question, thanks for starting the conversation! I'm generally pretty hesitant to allow customization of resources used for conformance tests. I think one of the most important goals of conformance tests is to show that different implementations are capable of handling the same set of configuration in the same way. If users need to make little tweaks here and there depending on the implementation, it makes the API less portable.

If at all possible, I'd try to find a way for your implementation to work without any modifications to the config. If not, I'd be interested in the specific limitations that prevent that from being possible, and if we can think of any workarounds.