WIP SRUopener #682
Draft
WIP SRUopener #682
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Nice seems to work. +1 |
TobiasNx
approved these changes
Mar 28, 2025
|
@dr0i is still in review? |
|
As we found out in #510 this PR needs a complete redesign. |
|
@TobiasNx can you do functional tests before I go on here? Have a look at the |
|
@dr0i I tried to install the dist: https://metafacture.github.io/metafacture-documentation/docs/flux/Flux-User-Guide.html#build-from-local-distribution to try the runner for functional testing but it runs into errors: When I test the flux.sh then it outputs the following: Can you help? (I tested the current master to compare, there |
|
Ah, I accidently removed the TarReader. |
|
Current version stucks in an endless SRU request loop starting by 1 again after finishing all request does not matter if a total number of records is given or not: e.g. Both result in, see that recordPosition 1 is turning up again after the expected last recordPosition 8: <?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
<record type="Authority">
<leader>00000nz a2200000nc 4500</leader>
<controlfield tag="001">042278333</controlfield>
<controlfield tag="003">DE-101</controlfield>
<controlfield tag="005">20110429135047.0</controlfield>
<controlfield tag="008">900305n||azznnaabn | ana |c</controlfield>
<datafield ind1="7" ind2=" " tag="024">
<subfield code="a">4227833-8</subfield>
<subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
<subfield code="2">gnd</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="035">
<subfield code="a">(DE-101)042278333</subfield>
...
<datafield ind1=" " ind2=" " tag="913">
<subfield code="S">swd</subfield>
<subfield code="i">k</subfield>
<subfield code="a">Internationaler Sozialistenkongress</subfield>
<subfield code="0">(DE-588c)4021089-3</subfield>
</datafield>
</record>
</collection></recordData><recordPosition>8</recordPosition></record></records><echoedSearchRetrieveRequest><version>1.1</version><query>WOE=sozialistenkongress and COD=s</query><xQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><startRecord>6</startRecord><maximumRecords>5</maximumRecords><recordSchema>MARC21plus-xml</recordSchema></echoedSearchRetrieveRequest></searchRetrieveResponse>
<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
<record type="Authority">
<leader>00000nz a2200000nc 4500</leader>
<controlfield tag="001">042278333</controlfield>
<controlfield tag="003">DE-101</controlfield>
<controlfield tag="005">20110429135047.0</controlfield>
<controlfield tag="008">900305n||azznnaabn | ana |c</controlfield>
<datafield ind1="7" ind2=" " tag="024">
<subfield code="a">4227833-8</subfield>
<subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
<subfield code="2">gnd</subfield>
...
</datafield>
<datafield ind1=" " ind2=" " tag="913">
<subfield code="S">swd</subfield>
<subfield code="i">c</subfield>
<subfield code="a">Bern / Internationaler Sozialistenkongress <1919></subfield>
<subfield code="0">(DE-588c)4227833-8</subfield>
</datafield>
</record>
</collection></recordData><recordPosition>1</recordPosition></record><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
<record type="Authority">
<leader>00000nz a2200000nc 4500</leader>
<controlfield tag="001">1267605979</controlfield>
<controlfield tag="003">DE-101</controlfield>
<controlfield tag="005">20230329111229.0</controlfield>
<controlfield tag="008">220908n||azznnaabn | ana |c</controlfield>
<datafield ind1="7" ind2=" " tag="024">
<subfield code="a">1267605979</subfield>
<subfield code="0">http://d-nb.info/gnd/1267605979</subfield>
<subfield code="2">gnd</subfield>
... |
TobiasNx
requested changes
Jun 20, 2025
There was a problem hiding this comment.
It seems that SRU opener stucks in infinite loop. See: #682 (comment)
Both now logs only one object batches and only 8 records are fetched. This is good! |
|
One think that just came to my mind is that the sru opener needs the provide to provide a user agent |
|
Code review: @fsteeg |
|
This seems to be the same issue as in #694 DetailsIt seems that there is some kind of broken xml that is passed through: This creates exzessive indetations. e.g.: Using list-fix-paths hints that there seems to be something wrong with the incoming xml: Exception: |
|
@TobiasNx Can you update here that this is no bug but caused by marcxmlplus (or so) ? |
|
@dr0i The behaviour I reported was not the problem from the workshop. But the identation behaviour i reported is related to |
Every single output is a valid XML by itself.
|
Hi @blackwinter if you have some time: can you implement tests here? Functional-wise is the modul ready. |
blackwinter
requested changes
Jul 22, 2025
Comment on lines
+22
to
+23
| api project(':metafacture-formatting') | ||
| api project(':metafacture-xml') |
There was a problem hiding this comment.
Why are these added dependencies needed?
| @@ -0,0 +1,234 @@ | |||
| /* Copyright 2013 Pascal Christoph. | |||
| * Licensed under the Eclipse Public License 1.0 */ | |||
There was a problem hiding this comment.
Why the different license? Metafacture is APL.
| private int maximumRecords = MAXIMUM_RECORDS; | ||
| private int startRecord = START_RECORD; | ||
| private int totalRecords = Integer.MAX_VALUE; | ||
| int numberOfRecords = Integer.MAX_VALUE; |
There was a problem hiding this comment.
Why is this variable package-private?
| * | ||
| * @param totalRecords total number of records to be retrieved | ||
| */ | ||
| public void setTotal(final String totalRecords) { |
There was a problem hiding this comment.
Why are these String setters instead of int?
| DocumentBuilder docBuilder = factory.newDocumentBuilder(); | ||
| Document xmldoc = docBuilder.parse(inputStreamOfURl); | ||
|
|
||
| Transformer t = TransformerFactory.newInstance().newTransformer(); |
| int nextRecordPosition = getIntegerValueFromElement(xmldoc,"nextRecordPosition", totalRecords); | ||
|
|
||
| recordsRetrieved = recordsRetrieved + nextRecordPosition - recordPosition; | ||
| startRecord = nextRecordPosition; // grenzwert : wenn maximumRcords > als in echt |
There was a problem hiding this comment.
Translate comment, fix typo.
|
|
||
| private InputStream retrieveUrl(StringBuilder srUrl, int startRecord, int maximumRecords) throws IOException { | ||
| final URL urlToOpen = | ||
| new URL(srUrl.toString() + "&maximumRecords=" + maximumRecords + "&startRecord=" + startRecord); |
There was a problem hiding this comment.
maximumRecords is invariant and could be set on the base URL (srUrl).
Comment on lines
+64
to
+68
| int offset = 0; | ||
| for (int i = 0; i < size; ++i) { | ||
| resultCollector.append(buffer, offset, size - offset); | ||
| offset = i + 1; | ||
| } |
There was a problem hiding this comment.
What is this loop supposed to do? It seems to append all suffixes of buffer to resultCollector, thus resulting in an OOM in test().
Suggested change
| int offset = 0; | |
| for (int i = 0; i < size; ++i) { | |
| resultCollector.append(buffer, offset, size - offset); | |
| offset = i + 1; | |
| } | |
| resultCollector.append(buffer, 0, size); |
|
|
||
| while (!stopRetrieving && recordsRetrieved < totalRecords && (startRecord < numberOfRecords)) { | ||
| InputStream inputStream = getXmlDocsViaSru(srUrl); | ||
| getReceiver().process(new InputStreamReader(inputStream)); |
There was a problem hiding this comment.
What are the repercussions of passing each page as an individual document to downstream consumers? Would it make more sense to pass either the result as a whole, or each record individually?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a draft and WIP.
@TobiasNx you can use it for functional testing.
Resolves #510.