get_element() fails to find XSD element when path contains indexed suffixes starting from [2] - Expected behavior or regression?

[serwizz]

Hello,

I've encountered an issue with the get_element() method where it can find elements when the path contains the index [1], but fails and returns None for any higher indexes like [2], [3], etc.

I'm not entirely sure if this is expected behavior or a regression, as the documentation doesn't explicitly specify how indexed XPath suffixes should be handled. However, this behavior is problematic for our use case where we automatically generate XPaths from XML documents using lxml.etree.getpath(), which includes these indexes starting from [1] for all elements in a sequence.

Steps to Reproduce:

Have an XSD schema with a sequence of elements where the same name appears multiple times (e.g., via maxOccurs="unbounded").

Generate an XPath for elements in a corresponding XML file using lxml.etree.getpath().

Try to get the XSD element using schema.get_element(tag, path=path).

The method works for paths with [1] but returns None for paths with [2], [3], etc.

Minimal Reproducible Example:

import re

import xmlschema
from lxml import etree

xsd_path = '/path/to/test.xsd'
xml_path = '/path/to/test.xml'
schema = xmlschema.XMLSchema11(xsd_path)

xml_document = xmlschema.XmlDocument(xml_path, schema=schema)
xml_resource = xmlschema.XMLResource(source=xml_document)
namespaces = xml_resource.get_namespaces()

tree = etree.parse(xml_path)
for xml_element in tree.iter():
    tag = xml_element.tag
    path: str = tree.getpath(xml_element)

    # This call works for [1] but fails for [2], [3], etc.
    xsd_element = schema.get_element(
        tag,
        path=path,
        namespaces=namespaces,
    )
    if xsd_element is None:
        print(f'FAILED: XSD element not found for path="{path}", tag="{tag}"')
    else:
        print(f'SUCCESS for path: "{path}"')

    # Workaround: remove indexes
    cleaned_path = re.sub(r'\[\d+\]', '', path)
    xsd_element_by_cleaned_path = schema.get_element(
        tag,
        path=cleaned_path,
        namespaces=namespaces,
    )
    if xsd_element is None and xsd_element_by_cleaned_path is not None:
        print(f'SUCCESS with cleaned path: "{cleaned_path}"')

Test files:

• test.xml - https://www.paste.org/129573
• test.xsd - https://www.paste.org/129574

Actual Output:

Based on my test with a catalog of books, here's the actual output demonstrating the issue:

SUCCESS for path: "/catalog"
SUCCESS for path: "/catalog/book[1]"
SUCCESS for path: "/catalog/book[1]/title"
SUCCESS for path: "/catalog/book[1]/author"
SUCCESS for path: "/catalog/book[1]/year"
SUCCESS for path: "/catalog/book[1]/price"
SUCCESS for path: "/catalog/book[1]/category"
FAILED: XSD element not found for path="/catalog/book[2]", tag="book"
SUCCESS with cleaned path: "/catalog/book"
FAILED: XSD element not found for path="/catalog/book[2]/title", tag="title"
SUCCESS with cleaned path: "/catalog/book/title"
FAILED: XSD element not found for path="/catalog/book[2]/author", tag="author"
SUCCESS with cleaned path: "/catalog/book/author"
FAILED: XSD element not found for path="/catalog/book[2]/year", tag="year"
SUCCESS with cleaned path: "/catalog/book/year"
FAILED: XSD element not found for path="/catalog/book[2]/price", tag="price"
SUCCESS with cleaned path: "/catalog/book/price"
FAILED: XSD element not found for path="/catalog/book[2]/category", tag="category"
SUCCESS with cleaned path: "/catalog/book/category"
FAILED: XSD element not found for path="/catalog/book[3]", tag="book"
SUCCESS with cleaned path: "/catalog/book"
FAILED: XSD element not found for path="/catalog/book[3]/title", tag="title"
SUCCESS with cleaned path: "/catalog/book/title"
FAILED: XSD element not found for path="/catalog/book[3]/author", tag="author"
SUCCESS with cleaned path: "/catalog/book/author"
FAILED: XSD element not found for path="/catalog/book[3]/year", tag="year"
SUCCESS with cleaned path: "/catalog/book/year"
FAILED: XSD element not found for path="/catalog/book[3]/price", tag="price"
SUCCESS with cleaned path: "/catalog/book/price"
FAILED: XSD element not found for path="/catalog/book[3]/category", tag="category"
SUCCESS with cleaned path: "/catalog/book/category"

Questions:

Is this the expected behavior? Should get_element() only work with [1] but not with higher indexes?

If this is expected, what is the recommended approach to handle XPaths generated by lxml.etree.getpath() that include these indexes?

If this is a regression, could it be fixed to handle all indexes consistently?

Current workaround:

We're currently using a regex to remove all [\d+] substrings from the path before calling get_element(), which works but feels like a workaround rather than a proper solution.

Environment:

• Python Version: 3.11
• xmlschema Version: 4.2.0
• lxml Version: 5.2.2

Additional context:

I suspect this might be a regression because our code worked with an older version of xmlschema 3.4.5.

[brunato]

Hi,

this is related to #454, as I answered for that issue:

this could be an issue or not. It depends on different structure of schemas and XML data. Maybe I can provide a more informative message if only path argument is provided and XPath on schema selects nothing.
In alternative a derived XPath parser can be used, like is already done for XSD identities and assertions, trying to redefine XPath predicate selector to accomplish an extended selection of schema nodes, with a fallback on selecting at least one schema node.

This is not a regression. The problem of simply ignore context position is to select a node when the right behavior is to not select (see test case test_xpath.XMLSchemaXPathTest.test_xpath_group). The workaround of removing predicate positions is not a solution for all the cases.

So I will try to solve predicate selection on schemas implementing a custom parser, that is more flexible and maintainable.
The new custom parser will be applied only to find/findall/iterfind APIs on schemas. These APIs are not part of XML Schema standard like XPath selectors for assertions or identity constraints.

The custom parser will redefine the predicate expression with a selection rule for numeric position predicate that selects the node if one of the following rules is true:

1. predicate value is equal to context position (that is the XPath rule)
2. context size is equal to 1 and predicate is an integer value greater than 1

this is the code chunk that will change:

        if len(predicate) == 1 and isinstance(predicate[0], NumericProxy):
            if context.position == predicate[0]:
                yield context.item
            elif context.size == 1 and isinstance(predicate[0], int) and predicate[0] > 1:
                yield context.item
        elif self.boolean_value(predicate):
            yield context.item

thanks

[brunato]

The release v4.3.0 try to solve this issue. Check it out when you can.