Export API watch history results as formatted json

[Taomyn]

I have a successfully working watch that's using Playwright to grab 4 fields of data, the filters were created from the visual selector - one is text the other three are numerical values.

When I use API to grab the latest snapshot all I get are results as a few lines of text.

Is there a way for the API to return the data as formatted json? I need to be able grab the data with another application that import either json or xml.

[dgtlmoon]

When I use API to grab the latest snapshot all I get are results as a few lines of text.

it's not clear if this is correct or not? what are you trying to achieve exactly? can you be more precise with some examples? its hard to follow

Is there a way for the API to return the data as formatted json? I need to be able grab the data with another application that import either json or xml.

So are you saying that it works, but its in CSV although you want it in JSON or? can you provide an example how that JSON might look? it is JSON isnt it? or not?

[Taomyn]

This is the URL I am watching: https://guichet.public.lu/en/entreprises/outils/prix-prod-petrol.html

This is the filter:

/html/body/div[5]/main/section[2]/div/div/div[3]/p
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[3]/td[2]
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[2]/td[2]
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[1]/td[2]

When I execute:

curl https://changedetection.mydomain.com/api/v1/watch/a-redacted-36/history/latest -H"x-api-key:d-redacted-9" -H "Content-Type: application/json"

It returns:

  Prices applicable from 6 August 2024.

  1.470 1.672 1.559

If I use the api call to list watches, it does return json output, so I think all is ok with my main installation.

[dgtlmoon]

Hmm you still didnt describe the solution tho, you want the snapshot or the history list or what as JSON? what is it exactly that you're to solve?

[dgtlmoon]

I'm trying to image what you could mean

/html/body/div[5]/main/section[2]/div/div/div[3]/p
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[3]/td[2]
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[2]/td[2]
/html/body/div[5]/main/section[2]/div/div/aside/div[1]/table/tbody/tr[1]/td[2]

maybe each result of each line is a entry in a json list?

[Taomyn]

Ok, sorry, it would be nice to have something like this returned, which would be the result of requesting all the history:

{"title":"Guichet Fuel Prices","description":"Latest Guichet Fuel Prices","pubDate":"2024-08-06T04:19:54.230-07:00","items":[{"Price95Fuel":"1.559","Price98Fuel":"1.672","PriceDieselFuel":"1.470","PricesChanged":"6 August 2024","guid":{"contents":12888,"isPermaLink":"false"},"pubDate":"Mon, 05 Aug 2024 18:00:08 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.672","PriceDieselFuel":"1.470","PricesChanged":"25 July 2024","guid":{"contents":12854,"isPermaLink":"false"},"pubDate":"Wed, 24 Jul 2024 18:00:02 -0700"},{"Price95Fuel":"1.571","Price98Fuel":"1.672","PriceDieselFuel":"1.488","PricesChanged":"20 July 2024","guid":{"contents":12834,"isPermaLink":"false"},"pubDate":"Fri, 19 Jul 2024 18:00:02 -0700"},{"Price95Fuel":"1.571","Price98Fuel":"1.672","PriceDieselFuel":"1.488","PricesChanged":"19 July 2024","guid":{"contents":12832,"isPermaLink":"false"},"pubDate":"Thu, 18 Jul 2024 18:00:04 -0700"},{"Price95Fuel":"1.585","Price98Fuel":"1.699","PriceDieselFuel":"1.508","PricesChanged":"13 July 2024","guid":{"contents":12806,"isPermaLink":"false"},"pubDate":"Fri, 12 Jul 2024 18:00:08 -0700"},{"Price95Fuel":"1.600","Price98Fuel":"1.699","PriceDieselFuel":"1.508","PricesChanged":"12 July 2024","guid":{"contents":12804,"isPermaLink":"false"},"pubDate":"Thu, 11 Jul 2024 18:00:10 -0700"},{"Price95Fuel":"1.600","Price98Fuel":"1.699","PriceDieselFuel":"1.525","PricesChanged":"11 July 2024","guid":{"contents":12800,"isPermaLink":"false"},"pubDate":"Wed, 10 Jul 2024 18:00:01 -0700"},{"Price95Fuel":"1.600","Price98Fuel":"1.716","PriceDieselFuel":"1.525","PricesChanged":"05 July 2024","guid":{"contents":12780,"isPermaLink":"false"},"pubDate":"Thu, 04 Jul 2024 18:00:02 -0700"},{"Price95Fuel":"1.582","Price98Fuel":"1.695","PriceDieselFuel":"1.525","PricesChanged":"29 June 2024","guid":{"contents":12766,"isPermaLink":"false"},"pubDate":"Fri, 28 Jun 2024 18:00:06 -0700"},{"Price95Fuel":"1.582","Price98Fuel":"1.695","PriceDieselFuel":"1.525","PricesChanged":"28 June 2024","guid":{"contents":12762,"isPermaLink":"false"},"pubDate":"Thu, 27 Jun 2024 18:00:05 -0700"},{"Price95Fuel":"1.582","Price98Fuel":"1.695","PriceDieselFuel":"1.513","PricesChanged":"22 June 2024","guid":{"contents":12738,"isPermaLink":"false"},"pubDate":"Fri, 21 Jun 2024 18:00:05 -0700"},{"Price95Fuel":"1.582","Price98Fuel":"1.671","PriceDieselFuel":"1.513","PricesChanged":"21 June 2024","guid":{"contents":12732,"isPermaLink":"false"},"pubDate":"Thu, 20 Jun 2024 18:00:06 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.671","PriceDieselFuel":"1.513","PricesChanged":"19 June 2024","guid":{"contents":12724,"isPermaLink":"false"},"pubDate":"Tue, 18 Jun 2024 18:00:09 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.671","PriceDieselFuel":"1.489","PricesChanged":"14 June 2024","guid":{"contents":12700,"isPermaLink":"false"},"pubDate":"Thu, 13 Jun 2024 18:00:06 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.671","PriceDieselFuel":"1.456","PricesChanged":"13 June 2024","guid":{"contents":12698,"isPermaLink":"false"},"pubDate":"Wed, 12 Jun 2024 18:00:02 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.671","PriceDieselFuel":"1.456","PricesChanged":"12 June 2024","guid":{"contents":12694,"isPermaLink":"false"},"pubDate":"Tue, 11 Jun 2024 18:00:07 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.648","PriceDieselFuel":"1.456","PricesChanged":"11 June 2024","guid":{"contents":12690,"isPermaLink":"false"},"pubDate":"Mon, 10 Jun 2024 18:00:10 -0700"},{"Price95Fuel":"1.559","Price98Fuel":"1.648","PriceDieselFuel":"1.442","PricesChanged":"07 June 2024","guid":{"contents":12682,"isPermaLink":"false"},"pubDate":"Sun, 09 Jun 2024 23:45:45 -0700"},{"Price95Fuel":"1.594","Price98Fuel":"1.694","PriceDieselFuel":"1.475","PricesChanged":"18/05/2024","guid":{"contents":12674,"isPermaLink":"false"},"pubDate":"Thu, 06 Jun 2024 23:52:31 -0700"},{"Price95Fuel":"1.594","Price98Fuel":"1.694","PriceDieselFuel":"1.475","PricesChanged":"18/05/2024","guid":{"contents":12610,"isPermaLink":"false"},"pubDate":"Sat, 18 May 2024 18:00:06 -0700"},{"Price95Fuel":"1.594","Price98Fuel":"1.711","PriceDieselFuel":"1.475","PricesChanged":"16/05/2024","guid":{"contents":12606,"isPermaLink":"false"},"pubDate":"Thu, 16 May 2024 18:00:08 -0700"},{"Price95Fuel":"1.611","Price98Fuel":"1.711","PriceDieselFuel":"1.498","PricesChanged":"11/05/2024","guid":{"contents":12584,"isPermaLink":"false"},"pubDate":"Sat, 11 May 2024 18:00:09 -0700"},{"Price95Fuel":"1.623","Price98Fuel":"1.711","PriceDieselFuel":"1.483","PricesChanged":"07/05/2024","guid":{"contents":12572,"isPermaLink":"false"},"pubDate":"Tue, 07 May 2024 18:00:06 -0700"},{"Price95Fuel":"1.623","Price98Fuel":"1.748","PriceDieselFuel":"1.483","PricesChanged":"04/05/2024","guid":{"contents":12566,"isPermaLink":"false"},"pubDate":"Sat, 04 May 2024 18:00:07 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.748","PriceDieselFuel":"1.514","PricesChanged":"03/05/2024","guid":{"contents":12564,"isPermaLink":"false"},"pubDate":"Fri, 03 May 2024 18:00:04 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.800","PriceDieselFuel":"1.514","PricesChanged":"26/04/2024","guid":{"contents":12550,"isPermaLink":"false"},"pubDate":"Sat, 27 Apr 2024 06:00:05 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.776","PriceDieselFuel":"1.514","PricesChanged":"23/04/2024","guid":{"contents":12538,"isPermaLink":"false"},"pubDate":"Tue, 23 Apr 2024 18:00:04 -0700"},{"Price95Fuel":"1.674","Price98Fuel":"1.798","PriceDieselFuel":"1.514","PricesChanged":"20/04/2024","guid":{"contents":12532,"isPermaLink":"false"},"pubDate":"Sat, 20 Apr 2024 18:00:11 -0700"},{"Price95Fuel":"1.674","Price98Fuel":"1.798","PriceDieselFuel":"1.565","PricesChanged":"18/04/2024","guid":{"contents":12528,"isPermaLink":"false"},"pubDate":"Thu, 18 Apr 2024 18:00:02 -0700"},{"Price95Fuel":"1.674","Price98Fuel":"1.811","PriceDieselFuel":"1.565","PricesChanged":"17/04/2024","guid":{"contents":12520,"isPermaLink":"false"},"pubDate":"Wed, 17 Apr 2024 18:00:10 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.811","PriceDieselFuel":"1.565","PricesChanged":"12/04/2024","guid":{"contents":12506,"isPermaLink":"false"},"pubDate":"Fri, 12 Apr 2024 18:00:11 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.886","PriceDieselFuel":"1.565","PricesChanged":"11/04/2024","guid":{"contents":12504,"isPermaLink":"false"},"pubDate":"Thu, 11 Apr 2024 18:00:02 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.886","PriceDieselFuel":"1.599","PricesChanged":"05/04/2024","guid":{"contents":12498,"isPermaLink":"false"},"pubDate":"Wed, 10 Apr 2024 23:18:56 -0700"},{"Price95Fuel":"1.653","Price98Fuel":"1.886","PriceDieselFuel":"1.599","PricesChanged":"","guid":{"contents":12496,"isPermaLink":"false"},"pubDate":"Wed, 10 Apr 2024 21:24:45 -0700"},{"Price95Fuel":"1.604","Price98Fuel":"1.829","PriceDieselFuel":"1.557","PricesChanged":"","guid":{"contents":12450,"isPermaLink":"false"},"pubDate":"Mon, 25 Mar 2024 18:00:03 -0700"},{"Price95Fuel":"1.604","Price98Fuel":"1.829","PriceDieselFuel":"1.574","PricesChanged":"","guid":{"contents":12442,"isPermaLink":"false"},"pubDate":"Wed, 20 Mar 2024 18:00:08 -0700"},{"Price95Fuel":"1.604","Price98Fuel":"1.829","PriceDieselFuel":"1.552","PricesChanged":"","guid":{"contents":12434,"isPermaLink":"false"},"pubDate":"Tue, 19 Mar 2024 18:00:06 -0700"},{"Price95Fuel":"1.573","Price98Fuel":"1.797","PriceDieselFuel":"1.552","PricesChanged":"","guid":{"contents":12394,"isPermaLink":"false"},"pubDate":"Fri, 01 Mar 2024 18:00:09 -0800"},{"Price95Fuel":"1.573","Price98Fuel":"1.783","PriceDieselFuel":"1.578","PricesChanged":"","guid":{"contents":12388,"isPermaLink":"false"},"pubDate":"Thu, 29 Feb 2024 18:00:09 -0800"},{"Price95Fuel":"1.559","Price98Fuel":"1.783","PriceDieselFuel":"1.578","PricesChanged":"","guid":{"contents":12362,"isPermaLink":"false"},"pubDate":"Thu, 22 Feb 2024 18:00:05 -0800"}]}

[Taomyn]

Sorry to revisit this after so long, but I have tried again and realised my explanation was not great. What I was originally after was to get the scrape to export the result as json file stored somewhere when it changed that could be downloaded, but if that's not possible then I would need to find a way to call the API (I'm using the data in a few places e.g. Zabbix and Home Assistant).

What I was hoping when I called the api to get the latest scrape result with something like:

curl https://changedetection.mydomain.com/api/v1/watch/<redacted>/history/latest -H "x-api-key:<redacted>" -H "Content-Type: application/json"

I would get back:

[{"0":"Prices applicable from 03 May 2025","1":"1.434","2":"1.536","3":"1.345"}]

Rather than:

  Prices applicable from 03 May 2025

  1.434 1.536 1.345

Does that make sense? I just used 0,1,2,3 for the field IDs as an example, but anything more appropriate would be fine that allows me to parse the result into the right places.