[bugfix] fn:transform: Conversion and treeIndex problems #5680

nverwer · 2025-03-18T19:53:24Z

Description

The fn:transform function sometimes produced NullPointerExceptions. This happened for documents containing nodes of type org.exist.dom.memtree.ElementImpl and other subtypes of org.exist.dom.memtree.NodeImpl.

The function org.exist.xquery.functions.fn.transform.Convert.ToSaxon.ofNode(Node) uses org.exist.xquery.functions.fn.transform.TreeUtils.treeIndex(Node) to get a 'path' (as a list of indexes) of the Node in its containing document. This 'path' is used to find the node in a newly built document that is suitable for use by Saxon.
Using the 'path' that the treeIndex function produces assumes that it starts at the document node.

However, some nodes do not have a containing document. Specifically, in org.exist.dom.memtree.NodeImpl.getParentNode():

if (parent.getNodeType() == DOCUMENT_NODE && !((DocumentImpl) parent).isExplicitlyCreated()) {
            /*
                All nodes in the MemTree will return an Owner document due to how the MemTree is implemented,
                however the explicitlyCreated flag tells us whether there "really" was a Document Node or not.
                See https://github.com/eXist-db/exist/issues/1463
             */
            return null;

In this case, the 'path' returned by treeIndex does not contain the index (0) for the root node, and org.exist.xquery.functions.fn.transform.TreeUtils.xdmNodeAtIndex(XdmNode, List<Integer>) will return null when a node is expected.

The proposed fix tests for this case. Going up the ancestor chain, when a node that is not a document node, and that does not have a parent is found, the index 0 for this root element is inserted in the 'path'.

Testing

The NullPointerException happened in a rather complicated XQuery with documents composed from several sources.
I have not yet been able to make a simple test case, but I will keep trying.

The Java code of DirectoryList throws an IllegalStateException when a non-existent directory is supplied as its first argument, which is an unchecked exception and therefore not caught in Java.

…op-6.x.x

…into develop-6.x.x

see eXist-db#5450 close eXist-db#4151 close eXist-db#5532 close eXist-db#4153

set :release to develop-6.x.x branch

[6.x.x] Repair JNLP interface, use correct BC library

- load the module from existdb instead of an external source - test with and without xmldb: scheme - fix template under test to actually do something - ensure that the output is checked for correctness

[6.x.x] Backport eXist-db#5574

When importing stylesheets the URIs are resolved in order - registered import-uris in package repo - XMLDB-locations (relative, absolute and with or without scheme) - any other location is treated as an exteranal source The test class and its cases were renamed to reflect their purpose. Some inline comments were added to help describing the intent.

fixes eXist-db#5525 fixes eXist-db#5530 - add functx to autodeploy for xquery tests - add tests for one-off queries with module imports - of a registered module without location hint - of a module with location hint - change XQueryContext to allow imports again - change SourceFactory to work with contextPath set to "."

see fabric8io/docker-maven-plugin#1289

keep :latest and :release tags aligned add dev:6 tag for snapshots from develop-6.x.x see eXist-db#3953

Backport multi arch 5533

[6.x.x] allow module imports in one-off xqueries

Closes eXist-db/exist#5680

…rent

Closes eXist-db/exist#5680

duncdrum · 2025-08-26T07:34:51Z

@nverwer thank you for the time you spend on this. We would like to include this PR in a patch release of exist. Could you rebase the PR please so, we can start with a cleaner commit history? I will try to help you with adding a test case. Did you succeed in the meantime in wiggling things down? Could you share the xsl that triggered the NPE?

nverwer · 2025-08-28T07:22:02Z

@nverwer Could you rebase the PR please so, we can start with a cleaner commit history?

What should I rebase this on? EDIT: I see, it is develop-6.x.x.

About a test case: We had the problem several months ago in a very large application. I could not make a concise test case back then, and it would be even more difficult now. Designing a test case requires a thorough understanding of the internal representations of XML in eXist-db, which I do not have.

I apologize for not coming up with a useful test. I am aware that eXist can only advance by the efforts of the community around it.
My main concern is to keep our application running, and this PR has helped us to use fn:transform instead of transform:transform.

nverwer · 2025-08-28T08:31:22Z

After spending some time trying to rebase, it all seems to be terribly messed up, even resulting in automatically closing this PR. There are close to a 100 files with merge conflicts, most of which are totally unrelated to this PR. I have no idea why this went horribly wrong. I will give it one more try before giving up.

nverwer · 2025-08-28T08:50:45Z

I tried to apply the changes to develop-6.x.x and will reopen this PR.

duncdrum · 2025-08-28T09:00:55Z

@nverwer thank you, gave me quite a scare there for a minute

nverwer · 2025-08-28T09:17:50Z

@nverwer thank you, gave me quite a scare there for a minute
I should not have pushed prematurely. 😨
I could not reconstruct exactly what was in the original pull request, but it does include the fixes for the NPE. I added the code that allows maps and arrays to be passed as parameters to an XSL transformation. I probably should have made that into a separate pull request. 😬

nverwer · 2025-09-23T09:34:31Z

In the meantime, my version of develop-6.x.x evolved, and I pushed it. I seems that this pull request was botched as a result.
Since nothing has happened with this pull request, I propose that it is closed without merging.

nverwer · 2025-09-23T09:41:49Z

This PR became a mess after I changed my develop-6.x.x branch a few times, and these changes became visible here.
I got fed up with this PR hanging around without any progress, and closed it.
Then I decided to once more find the actual changes, and made a new PR: #5882

Nico Verwer and others added 30 commits September 15, 2023 16:36

fix for eXist-db#5028

55dbb6c

The Java code of DirectoryList throws an IllegalStateException when a non-existent directory is supplied as its first argument, which is an unchecked exception and therefore not caught in Java.

Merge remote-tracking branch 'evolvedbinary/develop-6.x.x' into devel…

2b35cf3

…op-6.x.x

Allow setting the namespace on result elements of sql:execute()

5ff847a

Merge branch 'develop-6.x.x' of https://github.com/evolvedbinary/exist …

6225431

…into develop-6.x.x

Merge branch 'develop-6.x.x' into jb-sql-executefunction

4fa7494

do not hang when database connection is not available

befae56

simple implementation of ixml using Markup Blitz

1c515e7

we cannot use fn yet

cf27783

move evaluation of return expression after type checking

d609601

Merge branch 'fix-5315-type_checking_order' into JB-current

3dd83fc

check URL validity in XQueryUrlRewrite

62a3579

set user in context in fn:apply, when needed

e41f58e

Merge branch 'develop-6.x.x' into JB-current

cbf9adc

[maven-release-plugin] prepare for next development iteration

ef0e86f

Merge remote-tracking branch 'origin/6.3.0-branch' into JB-current

e0ea2bc

[feature] add multi-arch images

ee5a8d9

see eXist-db#5450 close eXist-db#4151 close eXist-db#5532 close eXist-db#4153

[feature] multi arch for release

855cb3d

set :release to develop-6.x.x branch

[bugfix] Repair JNLP interface, use correct BC library

af2d67f

Merge pull request eXist-db#5554 from dizzzz/bugfix/port_of_5544_jnlp

5cbb4cc

[6.x.x] Repair JNLP interface, use correct BC library

[hotfix] fix and extend TransformFromPkgTest

07c2e87

- load the module from existdb instead of an external source - test with and without xmldb: scheme - fix template under test to actually do something - ensure that the output is checked for correctness

Merge pull request eXist-db#5575 from line-o/backport/5574

56551ac

[6.x.x] Backport eXist-db#5574

[fix] adjust container v6

5f12820

[fix] modify Docker file

c9b87d4

see fabric8io/docker-maven-plugin#1289

[ignore] deploy from ci

8e28333

[ignore] disable experimental

944ff2d

[fix] new tag dev6

95c16de

keep :latest and :release tags aligned add dev:6 tag for snapshots from develop-6.x.x see eXist-db#3953

Merge pull request eXist-db#5598 from duncdrum/backport-multi-arch-5533

71a82ca

Backport multi arch 5533

Merge pull request eXist-db#5586 from line-o/backport/5529

3011d1f

[6.x.x] allow module imports in one-off xqueries

adamretter added a commit to evolvedbinary/elemental that referenced this pull request Apr 25, 2025

[bugfix] Fix issues with Saxon to eXist-db XDM conversion and vice-versa

14845a7

Closes eXist-db/exist#5680

adamretter added a commit to evolvedbinary/elemental that referenced this pull request Apr 26, 2025

[bugfix] Fix issues with Saxon to eXist-db XDM conversion and vice-versa

9254f84

Closes eXist-db/exist#5680

adamretter added a commit to evolvedbinary/elemental that referenced this pull request Apr 26, 2025

[bugfix] Fix issues with Saxon to eXist-db XDM conversion and vice-versa

474de2a

Closes eXist-db/exist#5680

line-o added the needs develop port label May 6, 2025

Nico Verwer added 3 commits May 20, 2025 16:25

report db-connection pool info when threads are waiting

f53d890

Merge remote-tracking branch 'origin/release/exist-6.4.0' into JB-cur…

8713dd6

…rent

pom fixes

f8f9dd7

adamretter added a commit to evolvedbinary/elemental that referenced this pull request May 30, 2025

[bugfix] Fix issues with Saxon to eXist-db XDM conversion and vice-versa

7cc7686

Closes eXist-db/exist#5680

Allow maps and arrays as parameters in fn:transform

2c64411

line-o force-pushed the develop-6.x.x branch from 526b93e to 670af64 Compare July 7, 2025 07:06

dizzzz requested review from a team and DrRataplan July 24, 2025 09:25

duncdrum self-assigned this Aug 26, 2025

duncdrum added this to the eXist-6.4.1 milestone Aug 26, 2025

nverwer closed this Aug 28, 2025

nverwer force-pushed the develop-6.x.x branch from eaf0f0d to 261d81c Compare August 28, 2025 07:49

Merge branch 'develop-6.x.x' into pr5680

b196087

nverwer reopened this Aug 28, 2025

nverwer force-pushed the develop-6.x.x branch 2 times, most recently from ab0c76d to b196087 Compare September 23, 2025 09:31

nverwer closed this Sep 23, 2025

nverwer mentioned this pull request Sep 23, 2025

[bugfix] fn:transform: Conversion and treeIndex problems #5882

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[bugfix] fn:transform: Conversion and treeIndex problems #5680

[bugfix] fn:transform: Conversion and treeIndex problems #5680

Uh oh!

nverwer commented Mar 18, 2025

Uh oh!

duncdrum commented Aug 26, 2025

Uh oh!

nverwer commented Aug 28, 2025 •

edited

Loading

Uh oh!

nverwer commented Aug 28, 2025 •

edited

Loading

Uh oh!

nverwer commented Aug 28, 2025

Uh oh!

duncdrum commented Aug 28, 2025

Uh oh!

nverwer commented Aug 28, 2025

Uh oh!

nverwer commented Sep 23, 2025

Uh oh!

nverwer commented Sep 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

[bugfix] fn:transform: Conversion and treeIndex problems #5680

[bugfix] fn:transform: Conversion and treeIndex problems #5680

Uh oh!

Conversation

nverwer commented Mar 18, 2025

Description

Testing

Uh oh!

duncdrum commented Aug 26, 2025

Uh oh!

nverwer commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nverwer commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nverwer commented Aug 28, 2025

Uh oh!

duncdrum commented Aug 28, 2025

Uh oh!

nverwer commented Aug 28, 2025

Uh oh!

nverwer commented Sep 23, 2025

Uh oh!

nverwer commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nverwer commented Aug 28, 2025 •

edited

Loading

nverwer commented Aug 28, 2025 •

edited

Loading

nverwer commented Sep 23, 2025 •

edited

Loading