`ObjectReader.readValues` doesn't stream anymore since Jackson 3.0

[modular-music-and-code]

Search before asking

• I searched in the issues and found nothing similar.

Describe the bug

I have isolated a use case, see here (standalone Gradle project).

There are two unit tests, one for Jackson 2, one for Jackson 3.
They are mostly copy+paste except for the imports.
Streaming using ObjectReader.readValues works in Jackson 2 works, but fails in Jackson 3.

Version Information

3.0.1

Reproduction

( see the linked use case )
Run: >> gradle test

Expected behavior

Given following pseudo-code:

val m: ObjectMapper = /* create somehow */
val r: ObjectReader = m.reader()
val fileReader: BufferedReader = Files.newBufferedReader(file)
val p: JsonParser = r.createParser(fileReader)
val iter: Iterator<MyPOJO> = m.reader().readValues(p, MyPOJO::class.java)

The returned iter should iterate further while the underlying reader is read.
It does that with Jackson-2, but not with Jackson-3.

Additional context

No response

[pjfanning]

Please don't assume anyone will run your test projects. They are useful but can you spell out what the results are instead of using vague terms like 'fails'?

[modular-music-and-code]

Please don't assume anyone will run your test projects. They are useful but can you spell out what the results are instead of using vague terms like 'fails'?

thanks for the feedback, I added more info

[cowtowncoder]

@modular-music-and-code Thank you for reporting this, I'll have a look.

Ok, ideal way is to inline test case instead of full test project (easier to deal with from all devices), but this will work.

[cowtowncoder]

Hmmmh. Ok, I spoke too soon... this is somewhat more complicated than I anticipated.

But just to be sure: is the issue that

1. Functionally ObjectReader.readValues() works incorrectly (wrong bound values) or
2. Parser is being called in a way to read more tokens than necessary (possible all)

After (re)reading issue, glancing at repro, I think implication is 2. But I am not 100% sure of what this means:

The returned iter should iterate further while the underlying reader is read.
It does that with Jackson-2, but not with Jackson-3.

As to test, I think it should be possible to avoid actual Files & multi-threading just observing things at JsonParser state level... which is more useful wrt trying to see what is going on.

[modular-music-and-code]

2. Parser is being called in a way to read more tokens than necessary (possible all)

In the isolated test-case: I am guessing that the Parser attempt to read more than necessary for one POJO, therefor it blocks.

In the non-isolated real use-case (not present publicly), the parser now waits until the whole reader is finished, only then it shoots out all POJOs at once.

[cowtowncoder]

I am sorry, I find this reproduction bit hard to follow: it seems to me read just fails; never mind streaming?

org.opentest4j.AssertionFailedError: 
Expecting actual:
  ["write: MyPOJO(s=1)",
    "write: MyPOJO(s=2)",
    "write: MyPOJO(s=3)",
    "write: MyPOJO(s=4)",
    "write: MyPOJO(s=5)"]
to contain exactly (and in same order):
  ["write: MyPOJO(s=1)",
    " read: MyPOJO(s=1)",
    "write: MyPOJO(s=2)",
    " read: MyPOJO(s=2)",
    "write: MyPOJO(s=3)",
    " read: MyPOJO(s=3)",
    "write: MyPOJO(s=4)",
    " read: MyPOJO(s=4)",
    "write: MyPOJO(s=5)",
    " read: MyPOJO(s=5)"]
but could not find the following elements:
  [" read: MyPOJO(s=1)",
    " read: MyPOJO(s=2)",
    " read: MyPOJO(s=3)",
    " read: MyPOJO(s=4)",
    " read: MyPOJO(s=5)"]

It would seem better to first verify that functionally call succeeds before digging into streamability.

[modular-music-and-code]

I'll try to isolate the issue further the next 1-2 days.

[modular-music-and-code]

After simplifying the use-case, I found that there was no problem with ObjectReader.readValues, but instead Jackson-3 doesn't flush the underlying writer - so readValues didn't get a chance to read anything.

See here, again

[cowtowncoder]

@modular-music-and-code unless your code calls JsonGenerator.flush() (or close()), I don't see why it should, if this is the code:

			val m = mapper()
			val fileWriter = Files.newBufferedWriter(file, CREATE_NEW)
			val g = m.createGenerator(fileWriter)

			repeat(5) { counter ->
				val pojo = MyPOJO((1 + counter).toString())
				events += "write: $pojo"
				g.writePOJO(pojo)
				fileWriter.flush()
				sema.release()
				Thread.sleep(50)
			}

EDIT: ok code has changed. But basically g.writeXxx() won't (and shouldn't, IMHO) flush() by default. In 2.x it may have happened as a side-effect tho, as part of ObjectCodec.writeXxx() call.

So it is effective difference b/w 2 and 3 but not -- unless I missing something (possible!) -- a bug on its own, or something to change.

Am open to suggestions on Javadoc improvements etc.

[modular-music-and-code]

Shouldn't it flush because of enable(FLUSH_AFTER_WRITE_VALUE) (in the changed code) ?
For the old code - yes, that feature was not enabled.

Edit: now I saw in the library code that FLUSH_AFTER_WRITE_VALUE is enabled by default - so, the expectation that flush must occur after POJO-writes is correct after all?

[cowtowncoder]

SerializationFeature.FLUSH_AFTER_WRITE_VALUE is databind-level, so on ObjectMapper/ObjectWriter.writeValue(), yes. But JsonGenerator write methods are not part of databind API.
In a way this is a fix in 3.0: calls from JsonGenerator to databind avoid layer of overhead (basically, reuse SerializationContext) and use real context & state.
Former flushing was a side-effect of additional overhead, not a feature.

If you called ObjectWriter.writeValue() (and similar), feature would have effect; using lower-level JsonGenerator.writeXxx() methods, no.