Unexpected filter behaviour on some sites?

[freddieleeman]

Describe the bug
"No history found"

Version
dgtlmoon/changedetection.io:latest

To Reproduce
URL: https://www.digikey.nl/nl/products/detail/dfrobot/SEN0395/14322660
Fetch method: Both Playwright Chromium/Javascript
CSS/JSON/XPATH Filter: //div[@data-testid="price-and-procure-title"]

Expected behavior
Result: "0 op voorraad"

Additional context
Reverting to dgtlmoon/changedetection.io:0.39.14 resolves the issue

[dgtlmoon]

$ curl  https://www.digikey.nl/nl/products/detail/dfrobot/SEN0395/14322660|grep price-and-procure-ti
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  345k    0  345k    0     0   258k      0 --:--:--  0:00:01 --:--:--  257k

Doesnt appear to be issued from them..

[dgtlmoon]

With javascript turned off..

[freddieleeman]

That would explain the results without javascript. But with javascript, the result is the same:

[dgtlmoon]

They are doing something really clever somehow.. the xpath/css selector is in the tests and no one else reported an issue

I'm going to close this, BUT please please update with what you learn

its almost like they are using an iframe but not quite, they have something special here

[dgtlmoon]

When I use the filter generated by the visual selector tool it works

/html/body/div[2]/main/div/div[1]/div[2]/div/div/div[1]/div/div/span

and a 'wait before read' of 5 seconds

[freddieleeman]

How would you explain that reverting to dgtlmoon/changedetection.io:0.39.14 resolves the issue?

[dgtlmoon]

@freddieleeman unsure, maybe you can debug?

I can say the filter applies only sometimes, and ONLY for this website, and you are the only one to report it..

[dgtlmoon]

Can you explain why it works for me in the latest version?

[freddieleeman]

Here is another example:

https://1drv.ms/f/s!Ainhp1nBLzMJrEeyV5NOiKa8zDA3
Playwright Chromium/Javascript
//*[contains(@class, "nameplate")]

[dgtlmoon]

mm my xpath checker says nothing found

[freddieleeman]

That is weird. And this one?

https://nl.mouser.com/ProductDetail/DFRobot/SEN0395?qs=ljCeji4nMDmvEgq75EdCVA%3D%3D
//div[@class="row pdp-product-availability"]

[dgtlmoon]

@freddieleeman I think I found something interesting, could you try this change locally?

--- a/changedetectionio/content_fetcher.py
+++ b/changedetectionio/content_fetcher.py
@@ -303,7 +303,7 @@ class base_html_playwright(Fetcher):
             page = context.new_page()
             try:
                # Bug - never set viewport size BEFORE page.goto
-                response = page.goto(url, timeout=timeout * 1000, wait_until='commit')
+                response = page.goto(url, timeout=timeout * 1000, wait_until=None)

I think playwrights commit doesnt always work as expected

[freddieleeman]

No cigar; visual selector preview looks good, but no result.

[dgtlmoon]

I reopened this is as a discussion, i dont quite think it's a bug in the xpath/css selector, but there is something going on with the sites that we dont fully understand yet

[freddieleeman]

I've set up two docker containers, both :latest and :0.39.14 with the exact same configuration.

dgtlmoon/changedetection.io:latest

dgtlmoon/changedetection.io:0.39.14

URL: https://www.digikey.nl/nl/products/detail/dfrobot/SEN0395/14322660
Fetch method: Playwright Chromium/Javascript
Wait seconds: NULL
XPATH Filter: //div[@data-testid="price-and-procure-title"]
Webdriver: browserless/chrome:latest

[dgtlmoon]

@freddieleeman press the 'share' link and copy+paste the link it gives

[dgtlmoon]

@freddieleeman i got it to work on our hosted/subscription service, we have a bit more complicated fetching backend setup

https://changedetection.io/share/ESEzo5UVlWwa