Testing validations with a bad example

[xocasdashdash]

Hi there!

Please let me know if this is the wrong way of asking this quest as it's more of a support question than an issue with the project (been using it for 30 min and loving it already)

I'm trying to check how the validations work, so i have an api with an already valid example. I make a request to this endpoint and it works great (all using the mock mode).
Now i go and drop fields from the example that are marked as required in the api spec, at this point i'm expecting that the next request will show up as an error.

The reality is that is that it comes back ok. I've also tested by messing with the types (putting a string in a field that's defined as a bool). This also works ok.

I've been able to trigger the validations by making a bad request with a non valid body and i can see that it works as expected.

Is this the correct behaviour? i was hoping that if a server returns a "bad" response it would show up as a violation, is this because i'm running in mock mode?

[daveshanley]

Hi,

I am a little confused, when you drop fields from the example, where are you expecting it to fail? Examples are only used to render a response in mock mode, it's not used in any other way for validation, the schema is still what is used to validate requests and responses.

I need a little more information as to how you're using the tool (it's quite flexible and folks use it in lots of different ways)

[xocasdashdash]

yeah, so i'm trying to use as part of a local developer flow. The flow i'm thinking of is the following:

• Dev A writes a new API endpoint spec and adds some examples. How can he validate that the examples follow the spec?
• Assume that Dev A makes a mistake and doesn't check his examples correctly, how can Dev B use wiretap as a mock server to test the API if the specs are not validated it?

[daveshanley]

• Dev A writes a new API endpoint spec and adds some examples. How can he validate that the examples follow the spec?

I see, so this part of the puzzle is handled by another tool that does OpenAPI spec analysis. It's called vacuum and read more about it at https://quobix.com/vacuum

If you're feeling brave you can use our newest (early alpha) tool called the OpenAPI doctor, that will combine wiretap and vacuum into a suite (eventually) https://doctor.pb33f.io

I don't plan on building this functionality into wiretap, because the doctor will provide both wiretap capability and OpenAPI analysis and change detection.

[xocasdashdash]

ahh i understand, but do you think there's some difference in behaviour when you do a get request that returns an example, even though it's invalid. And when you do something like a post request that you can see a violation on the UI

[daveshanley]

ahh i understand, but do you think there's some difference in behaviour when you do a get request that returns an example, even though it's invalid. And when you do something like a post request that you can see a violation on the UI

I am struggling to understand what you mean, would you be able to explain it a little differently to me? I apologize, I work across multiple OpenAPI projects and my brain sees OpenAPI like a compiler does, I struggle a little sometimes with concepts like this.

[darrenspurgeon]

I was about to open another issue, but caught this similar comment about what I was looking to accomplish.

My use case or process:

• Have a valid spec, passes Vacuum lint.
• Testing against Wiretap's mock mode (wiretap -x)
• Running Hurl tests against Wiretap, just to demonstrate to engineers early test cases, show various example responses, and so on.
• Use Preferred header to return the examples I want.

All the above works beautifully. However, here's where I get a little stuck on testing non-200 responses.

I'm passing the wiretap-status-code header to return 422 where I want to return errors against several client requests with invalid requests. It does so if there are no specific validation problems in my request, but that's kind of where I get stuck.

For instance, let's say I want to show where a client passes a query param "status=Active" where status in the spec is an enum of values that does not include "Active". Now, that's a violation of the spec from the client point of view, though by this point the spec itself has passed the Vacuum lint because it properly defines the enum values. It's just the client test case doesn't.

What I'd like to do is have Wiretap recognize I've asked for a 422 (or just a non-200) response. Then, instead of returning a response message that the request violated the OpenAPI schema for that endpoint, I'd like it to look up an example (via Preferred header I pass) the spec example I put in the 422 response block. If I don't pass Preferred, then shame on me, and I'd expect the current behavior instead, meaning the spec violation message. Btw, this 422+example works if there are no client initiated spec violations.

I know it's kind of a stretch of what Wiretap is doing, but I think this could be useful in mock or proxy mode. It's kind of where I feel there's a gap between Wiretap and Vacuum in that regard. Maybe there's something I'm overlooking here as well.

[daveshanley]

I see,

So you want a bypass? As in, if no Preferred header results in the current behavior, but when it's present it will bypass the violation and still return the example, with the overridden code as well?

It might mean adding another header to request a bypass, so new behavior can be unlocked, without breaking the existing behavior.

[darrenspurgeon]

Yes, a bypass seems to fit the bill. I do like that Wiretap normally tells us when requests don't match the schema (are invalid), but I also want to show examples coming through 422 responses, for instance, when I intend to pass something incorrect. Thanks!