Error when decoding response from schema registry on request sent with header `Accept: application/vnd.schemaregistry.v1+json`

[andyrift]

Using avro and trying to use TopicNameStrategyWithSchema, schema validation fails with error could not parse to RawRegisteredSchema with cause error decoding response body
I tested with curl and the problem appears to be that we allow server to send data in format vnd.schemaregistry.v1+json but then try to only parse json

I can't really show screenshots or exact examples because corporate.
But I debugged with curl and narrowed the problem down to what is in this issue's name.

The exact line with the problem:

schema_registry_converter/src/async_impl/schema_registry.rs

Line 510 in 6cdc393

.header(ACCEPT, "application/vnd.schemaregistry.v1+json"),

So this

            .header(ACCEPT, "application/vnd.schemaregistry.v1+json"),

Needs to be this

            .header(ACCEPT, "application/json"),

I applied the fix locally and it solved the problem.

[gklijs]

Is that a client side or server side error? If it's a service side error, which kind of schema registry are you using? This is the correct one for the Confluent Schema Registry, if for some other one it needs to be the above, we can make it configurable. It likely needs to become part of the SrSettings in that case.

[andyrift]

More specifically, I found with curl that response to subjects/{subject}/versions post request with schema in the body is ConfluentSchemaId{id=1000} (Same when removing +json entirely).
If I set header to only accept json, I receive {"id":1000} as expected

Note: 1000 represents redacted value

[andyrift]

I actually don't know what registry we are using, I only know that in documentation there are links to confluent api reference
Is this sort of behavior familiar to you anyway?

[andyrift]

Maybe it is a problem in the registry, because removing Accept header entirely also solves this.

[andyrift]

You are right about correct request, I found this in documentation about content types: https://docs.confluent.io/platform/current/schema-registry/develop/api.html#content-types

Maybe it is worth adding those options for less specific content types to this projects?

[gklijs]

It helps to know what is exactly need for what registry, so I both know it's going to work, and I can add some documentation when you need something else than the defaults. The integration test is using the Confluent Schema registry. But it should not be that much trouble to add one for another registry. If this was a generic problem, it would have popped up sooner. I wouldn't mind implementing the needed changes, but I do need more information.

[andyrift]

I think it might be a local project or just some very old version. I am trying to figure it out through our support team.
Regarding this project, in the same documentation I linked earlier there is a section about content negotiation and in there confluent propose this possible request header: Accept: application/vnd.schemaregistry.v1+json; q=0.9, application/json; q=0.5 quote: "which can be useful when, for example, a new version of the API is preferred but you cannot be certain it is available yet."
I tried and it also solves the problem, the server sends the correct json

[gklijs]

I'm hesitant to change it, as I might break things for others, also I like to keep it as much the same as possible as the Java client, which does use the same header, https://github.com/confluentinc/schema-registry/blob/265441066d1ec64fe3481aec1c10700f583a95b6/client/src/main/java/io/confluent/kafka/schemaregistry/client/rest/Versions.java#L24 although in the PRERERRED_RESPONSE_TYPES is does have two others as well. Although the RUST confluent client put the header as CONTENT_TYPE and not ACCEPT https://github.com/rayokota/rust-schema-registry-client/blob/8621eda40b3c73ddb88ecd6d1d8643ef7e754028/src/rest/rest_service.rs#L79. So maybe the whole accept header can be omitted. @rayokota maybe you can help out which headers should be set?

[andyrift]

I am returning with yet more info
We are indeed using confluent schema registry, and the one I am hitting is version 6.2.2, I checked that it is the one with my schema
I received response from support team regarding the problem specifically and they said that and I quote, all three headers should be used as in this example https://docs.confluent.io/platform/current/schema-registry/develop/api.html#post--subjects-(string-%20subject)
It is not exactly the route used in this library and one I was testing, that is the one above it on the reference page, but close enough

[gklijs]

For the accept header that indeed does seem to be the case, the phyton cliënt does it also like that, https://github.com/marcosschroh/python-schema-registry-client/blob/8e3d7bc75ae9615795cfcc8295ddda7a1dbd2825/schema_registry/client/utils.py#L35. I do find it a bit weird that it only know pops up, it doesn't seem to be a general problem. Also, in the document shared, they say just one should be enough. Isn't is always json that is returned, or maybe some of the newer features of schema registry is used, that breaks it?

[andyrift]

I tested most of the routes and only this one, that is subjects/{subject}/versions, behaves this way, returning different response based on accept header. Even subjects/{subject} does not do this.
I will try to spin up 6.2.2 locally and test if it is always the case with this version, but I am not sure I will succeed :)

[gab-txt]

Hi, I don't know if related cause we have not yet pinned the issue down yet, but we started having this error since yesterday (could not parse to RawRegisteredSchema, error decoding response body). I'll write back once we have understood the cause better.

[andyrift]

So, I have the reason why this did not pop up sooner
I tested with different versions of registry and even with our modified images, could not replicate this weird behavior
I even found direct address of the registry I am using and it also works fine
So it is not the registry - that is why no one was having this trouble before
What I am having trouble with is a proxy to the registry. As far as I know, this proxy is there to have guardrails against unwanted modifications, and I suspect these guardrails might be screwing things up

[gklijs]

Might be, it's possible to define your own proxy, but I'm not sure if that can be used as a work around? Or change the configuration of the proxy?