Xpath filter is not consistent with xpather.com

[bladecell]

I am trying to watch this url, monitor car offers and return link to the offer.

I created xpath filter to extract the hrefs and verified it here and with this python code:

from lxml import html
import requests

# Define the URL
url = "https://www.tipcars.com/volkswagen-golf/hatchback/benzin/?2008-&-175000kc&-175000km&75-kw"

# Send a GET request to the URL
response = requests.get(url)

# Parse the HTML content
tree = html.fromstring(response.content)

# Use XPath to find all hrefs with the specified class
hrefs = tree.xpath("//a[contains(@class, 'float-l') and contains(@class, 'w-100')]/@href")

# Print the extracted hrefs
for href in hrefs:
    print(href)

The issue I am having is that ChangeDetection only retuns few href and not all like the python code and xpather and I have run out of ideas how to debug it.

This is the watch https://changedetection.io/share/pp78O8foFZUa

[Constantin1489]

Sorry for the noise. I tried and noticed various results I got. That is weird.
EDIT: the number 18 is the result of xpath:count(//a[contains(@class, 'float-l') and contains(@class, 'w-100')]/@href).
EDIT2: I got the number 24, which is the result of the script of yours.
EDIT3: this is xpath3's script which got 24 results.

from lxml import html, etree
import requests
from elementpath import select
from elementpath.xpath3 import XPath3Parser
parser = etree.HTMLParser()

# Define the URL
url = "https://www.tipcars.com/volkswagen-golf/hatchback/benzin/?2008-&-175000kc&-175000km&75-kw"

# Send a GET request to the URL
response = requests.get(url)

# Parse the HTML content
tree = html.fromstring(response.content, parser=parser)
res = select(tree, "//a[contains(@class, 'float-l') and contains(@class, 'w-100')]/@href", parser=XPath3Parser)

# Print the extracted hrefs
for href in res:
    print(href)

print(len(res))

Xpath3.1 - requests

Xpath3.1 - browser

xpath1 - browser

xpath1 - requests

[bladecell]

xpath:count(//a[contains(@class, 'float-l') and contains(@class, 'w-100')]/@href) and xpath1:count(//a[contains(@class, 'float-l') and contains(@class, 'w-100')]/@href) this for results in 6 for browser and requests. I cant even get the 18 links.
I have version v0.45.23

[Constantin1489]

I'm not joking but

diff --git a/changedetectionio/html_tools.py b/changedetectionio/html_tools.py
index a03653b..3c05ca6 100644
--- a/changedetectionio/html_tools.py
+++ b/changedetectionio/html_tools.py
@@ -122,7 +122,10 @@ def xpath_filter(xpath_filter, html_content, append_pretty_line_formatting=False
         # So that we can keep CDATA for cdata_in_document_to_text() to process
         parser = etree.XMLParser(strip_cdata=False)
 
-    tree = html.fromstring(bytes(html_content, encoding='utf-8'), parser=parser)
+    #tree = html.fromstring(bytes(html_content, encoding='utf-8'), parser=parser)
+    re_xml_encoding_declaration = re.compile(r'(?<=\<\?xml version)( *= *([\'\"])[^\1\ \n]*?\2 )encoding *= ?([\'\"])[^\1\n]*?\2 *').sub
+    html_content = re_xml_encoding_declaration(r'\1', html_content)
+    tree = html.fromstring(html_content, parser=parser)
     html_block = ""
 
     r = elementpath.select(tree, xpath_filter.strip(), namespaces={'re': 'http://exslt.org/regular-expressions'}, parser=XPath3Parser)

this patch will increase the number.

[Constantin1489]

I use this patch for the encoding problems but unintentionally it would partially fix this problem.(?)
BTW as I posted the xpath3 script, the problem happens before the even before the html.fromstring.

[Constantin1489]

This problem smells so funny. But I'm not sure I can prove "I solved"

[dgtlmoon]

Click the "sort text alphabetically" option, could help too

…

On 6 June 2024 14:59:07 UTC, Constantin Hong ***@***.***> wrote: Sorry for the noise. I tried and noticed various results I got. That is weird. Xpath3.1 - requests <img width="726" alt="Screenshot 2024-06-06 at 23 52 13" src="https://github.com/dgtlmoon/changedetection.io/assets/61624808/1deaae0f-ae52-4cfc-9983-53d42ecb79d0"> Xpath3.1 - browser <img width="966" alt="Screenshot 2024-06-06 at 23 53 00" src="https://github.com/dgtlmoon/changedetection.io/assets/61624808/a244b598-7df0-456c-b092-8dc8e9869a19"> xpath1 - browser <img width="939" alt="Screenshot 2024-06-06 at 23 54 01" src="https://github.com/dgtlmoon/changedetection.io/assets/61624808/e3d6e736-5852-4968-94d3-9b2d166abefe"> xpath1 - requests <img width="939" alt="Screenshot 2024-06-06 at 23 55 39" src="https://github.com/dgtlmoon/changedetection.io/assets/61624808/bff203a1-3f97-4ccf-a982-39345fb5ade5"> -- Reply to this email directly or view it on GitHub: #2401 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[bladecell]

I have that option enabled.