How do you extract a JSON property value from an HTTP source with unstructured JSON response?

[trevorstr]

• I launched Arroyo as a container
• I used the web interface to create an HTTP connect source with unstructured JSON schema response
• I tried to access the value field, but I just get an error saying it's an invalid field name

SELECT extract_json_string(value, '$.species.name') FROM http01;

When I created the HTTP connection source, it said that there would be a value field that I could access the JSON from. So how do I access the JSON response from the HTTP source, in my SELECT query?

[mwylde]

Hey @trevorstr — thanks for trying Arroyo!

If you expand the http01 source in the sources tab (clicking the v icon), it will show you the schema. If you created it with unstructured JSON, it should have a single value JSON field. If you see something else, I would try recreating the source and check that these options are selected:

[trevorstr]

@mwylde Thanks. Those are the exact options I selected, but there are no schema field entries. I created a duplicate HTTP source, but the same problem persisted.

[mwylde]

Hm, that's interesting. I'm not able to reproduce. What version of arroyo is this? (I.e., if you're running in docker, what's the tag of the image you're running?)

In the meantime you can work around this by creating the source in SQL instead of using the webui: (see the example at the bottom of https://doc.arroyo.dev/connectors/polling-http#configuring-the-connection):

create table http01 (
  value JSON
) with (
  connector = 'polling_http',
  endpoint = 'http://example.com',
  format = 'json',
  'json.unstructured' = 'true'
);

[trevorstr]

The container image hash part is 3fd812f4c149.

I just created another HTTP data source, and this time it has the value field. I have no idea why it was failing last night.