`OutOfMemoryError` while serving attachment from filesystem attachment provider

[avernet]

The following exception happens on 2024.1.2.

java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
    at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
    at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:132)
    at org.orbeon.io.IOUtils$.$anonfun$copyStreamAndClose$6(IOUtils.scala:37)
    at org.orbeon.io.IOUtils$$Lambda/0x000000000f81a7e0.apply$mcVI$sp(Unknown Source)
    at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1306)
    at org.orbeon.io.IOUtils$.$anonfun$copyStreamAndClose$2(IOUtils.scala:35)
    at org.orbeon.io.IOUtils$.$anonfun$copyStreamAndClose$2$adapted(IOUtils.scala:33)
    at org.orbeon.io.IOUtils$$Lambda/0x000000000f81ebf8.apply(Unknown Source)
    at org.orbeon.io.IOUtils$.useAndClose(IOUtils.scala:63)
    at org.orbeon.io.IOUtils$.$anonfun$copyStreamAndClose$1(IOUtils.scala:33)
    at org.orbeon.io.IOUtils$.$anonfun$copyStreamAndClose$1$adapted(IOUtils.scala:32)
    at org.orbeon.io.IOUtils$$Lambda/0x000000000f81e820.apply(Unknown Source)
    at org.orbeon.io.IOUtils$.useAndClose(IOUtils.scala:63)
    at org.orbeon.io.IOUtils$.copyStreamAndClose(IOUtils.scala:32)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD.$anonfun$get$1(FilesystemCRUD.scala:56)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD.$anonfun$get$1$adapted(FilesystemCRUD.scala:53)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD$Lambda/0x000000000fb227c0.apply(Unknown Source)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD.withFile(FilesystemCRUD.scala:103)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD.get(FilesystemCRUD.scala:53)
    at org.orbeon.oxf.fr.persistence.attachments.FilesystemCRUD.get$(FilesystemCRUD.scala:48)
    at org.orbeon.oxf.fr.persistence.attachments.Provider$Filesystem$.get(Provider.scala:24)
    at org.orbeon.oxf.fr.persistence.attachments.CRUDRoute$.process(CRUDRoute.scala:55)
    at org.orbeon.oxf.controller.PageFlowControllerProcessor$PageOrServiceRoute.process(PageFlowControllerProcessor.scala:656)
    at org.orbeon.oxf.controller.PageFlowControllerProcessor.start(PageFlowControllerProcessor.scala:245)
    at org.orbeon.oxf.pipeline.InitUtils$.runProcessor(InitUtils.scala:84)
    at org.orbeon.oxf.webapp.ProcessorService.$anonfun$service$1(ProcessorService.scala:47)
    at org.orbeon.oxf.webapp.ProcessorService$Lambda/0x000000000f4a7ac8.apply$mcV$sp(Unknown Source)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)

• IOUtils.scala.copyStreamAndClose is using a 8K buffer, which is fine. We see from the exception that the problem happens because we write in a ByteArrayOutputStream.
• This ByteArrayOutputStream is returned by LocalResponse.getOutputStream. It is a LocalByteArrayOutputStream extends ByteArrayOutputStream.
• We have a LocalResponse because the request comes from an InternalHttpClient, which creates a LocalResponse and passes it to LocalExternalContext.

+1 from customer

[ebruchez]

I am also getting an OutOfMemoryError when saving a 4GB attachment (JVM heap is 3GB).

[ebruchez]

When saving, we quickly reach the size of the heap, then fail.

[ebruchez]

When saving, the issue is this:

• putWithAttachments()
  • saveAttachmentIo()
    • Connection.connectAsync()
      • gets an fs2.Stream
      • ConnectionSupport.asyncToSyncStreamedContent()
        • which stores the stream into an Array[Byte]
        • returns IO[StreamedContent] from the passed AsyncStreamedContent

We have the following comment:

// Convert the fs2.Stream to an InputStream. Of course, We'd like to stream all the way ideally, but this is a
// first step. We cannot use fs2.io.toInputStream because it requires running two threads, which doesn't work in
// JavaScript. So we go through an in-memory Array for now. Note that sending data also works with Arrays.

Now, in the JVM env, we wouldn't have an issue with threads.

[ebruchez]

Right now, we have the following:

def fs2StreamToInputStreamInMemory(s: fs2.Stream[IO, Byte]): IO[InputStream] =
  s.compile.to(Array).map(new ByteArrayInputStream(_))

In Connection, connectWithContentIo() takes a StreamedContent (alias for StreamedContentT[InputStream]). This later ends up into LocalRequest.getInputStream.

Could a LocalRequest.getStream return an InputStream Either fs2.Stream? But then, would the processing model work out?

If we can't, then we should definitely not use a Array[Byte], but save the stream out of memory into a temporary file instead.

Now, on the other end:

• we reach the PersistenceProxy
• then FilesystemCRUD
• there, we call httpRequest.getInputStream

Could we do ok if we had instead an fs2.Stream here?

[ebruchez]

As a reminder, this is currently an issue only for the save action. That action already uses Stream extensively, which was done to support the JS environment, which is single-threaded.

What we would need to do is:

• being able to read an attachment, at least from a temporary file, as a Stream
  • Connection.connectAsync()
  • file: scheme
  • this only returns a Stream, but does not write one
  • P2: in general, so call to LocalRequest, etc.
• being able to write an attachment as a Stream
  • Connection.connectAsync()
  • LocalRequest
  • PersistenceProxy
  • FilesystemCRUD
  • this only writes a Stream, but does not read one

Note that if we want to support relational as well, in CreateUpdateDelete, before writing, we create an Array[Byte] again for the file_content column. As a first step, we could fix streaming only when using FilesystemCRUD.