ReadData and DataBlockCodec (#137)

* Add N5ReadBenchmark

* remove redundant modifiers

* WIP: remove @OverRide annotations for ByteBuffer methods

to be able to turn those methods on/off in the DataBlock interface
without causing compile errors in the implementations.

* Fix benchmark image URL

* Add byte[] DataType.createSerializeArray(numElements)

* WIP Compression / Codec API

* WIP encode/decode implementations

* bugfix

* cleanup

* wip

* feat: DataBlock methods to read/write directly from DataInput/Output

* WIP: SplittableReadData

* WIP DataBlock readData variants

* WIP SplittableReadData

* WIP clean up Compression interface

* WIP speed up DataBlock.readData(InputStream), and clean up

* WIP DataBlock.writeData(OutputStream)

* WIP use Splittable.ReadData in DefaultBlockReader

* WIP revise StringDataBlock (doesn't work yet)

* WIP revise StringDataBlock

revert to holding serialized and actual data. otherwise we can't be
compatible with existing N5 datasets.

* Add ByteOrder to DataBlock.writeData()

* Clean up

* Clean up

* fix javadoc error

* switch from byte[] to ByteBuffer for DataBlock.de/serialize()

* Add explicit commons-io dependency

* Lean more on ReadData

* Move ReadData etc to separate classes
* Add ReadData.writeTo(OutputStream)
* Add EncodedReadData that wraps a ReadData and an OutputStreamEncoder
* Compression (BytesCodec) can encode ReadData. (This might happen
  immediately or later when the ReadData is written to OutputStream).

* WIP ReadData.decode(Codec)

* Add decode(ReadData readData) variant that knows the length of the decoded data

* Hide ReadData implementations, use static ReadData factory methods instead

* Remove deprecated BlockWriter/BlockReader interfaces

* Clean up

* javacod

* Remove old Compression methods. Everything goes through ReadData

* Remove DataBlock de/serialize() methods

Maybe we'll rename readData()/writeData() later, the point is to remove
the ByteBuffer methods from the API

* fix javadoc error

* Add ReadData method toByteBuffer()

* clean up

* Use OutputStream wrapper to intercept close() in EncodedReadData

We still need a custom interface "OutputStreamOperator" because we want
to throw IOException and UnaryOperator::apply doesn't.

* ReadData.encode/decode methods forwarding to BytesCodec

* remove BytesCodec interface and put encode/decode into Compression for now

* idea

* Add LazyReadData and OutputStreamWriter

When data is requested from the LazyReadData, the LazyReadData will ask
its OutputStreamWriter to write the data to a ByteArrayOutputStream.

When the LazyReadData itself is written to an OutputStream, it will pass
that OutputStream to its OutputStreamWriter (without loading the data
into a byte[] array first).

* WIP Codecs

* WIP Codecs

* WIP Codecs

* WIP use Codecs, remove serialization methods from DataBlock

* Use ProxyOutputStream. FilterOutputStream is slow

* use only array type in DataBlockCodec generics

* refactor

* refactor

* convenience DataCodec methods to get codec with approprioate endianness

* typo

* refactor

* refactor

* Let DatasetAttributes provide the DataBlockCodec

avoids the Compression argument to encode/decode methods

* refactor, add ReadData.materialize()

This is in preparation for moving SplittableReadData into a separate PR

* Remove SplittableReadData interface

* fix javadoc error

* rename ChunkHeader to BlockHeader

* Remove unnecessary flush()s, instead close OutputStream where it is constructed

* Refactor for compatibility with upcoming Codecs Release (#2)

* refactor: don't pass `decodedLength` to ReadData

* refactor: move DataBlockFactory/DataBlockCodecFactory to N5Codecs

* feat: Add StringDataCodec, ObjectDataCodec,  StringDataBlockCodec, ObjectDataBlockCodec

DataCodecs now creat the access object and return it during `deserialize`

* refactor: add AbstractDataBlock to extract shared logic between Default/String/Object blocks

* refactor: DatasetAttributes responsible for DataBlockCodec creation

N5BlockCodec uses dataType (and potentially other DatasetAttributes to wrap the desired DataBlockCodec

* revert: add back createDataBlock logic in DataType

* doc: retain javadoc from before refactor

* revert: keep protected constructor with DataBlockCodec parameter

refactor: inline createDataBlockCodec from constructor params

* refactor: remove currently unused N5BlockCodec. Something like this may be needed when multiple codecs are supported

* refactor: dont expose N5Codecs internals

* refactor: rename encodeBlockHeader -> createBlockHeader

* feat: add ZarrStringDataCodec support

* ZarrStringDataCodec always LITTLE_ENDIAN

* fix javadoc

* clean up visibility and formatting

* bugfix

* improve readability

---------

Co-authored-by: John Bogovic <bogovicj@janelia.hhmi.org>
Co-authored-by: Caleb Hulbert <cmhulbert@users.noreply.github.com>

Author: 3 people

pom.xml

@@ -202,6 +202,22 @@
 			<groupId>org.apache.commons</groupId>
 			<artifactId>commons-compress</artifactId>
 		</dependency>
+		<dependency>
+			<groupId>commons-io</groupId>
+			<artifactId>commons-io</artifactId>
+		</dependency>
+
+		<!-- JMH -->
+		<dependency>
+			<groupId>org.openjdk.jmh</groupId>
+			<artifactId>jmh-core</artifactId>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.openjdk.jmh</groupId>
+			<artifactId>jmh-generator-annprocess</artifactId>
+			<scope>test</scope>
+		</dependency>
 	</dependencies>
 
 	<repositories>

src/main/java/org/janelia/saalfeldlab/n5/AbstractDataBlock.java

@@ -25,6 +25,8 @@
  */
 package org.janelia.saalfeldlab.n5;
 
+import java.util.function.ToIntFunction;
+
 /**
  * Abstract base class for {@link DataBlock} implementations.
  *
@@ -35,15 +37,21 @@
  */
 public abstract class AbstractDataBlock<T> implements DataBlock<T> {
 
-	protected final int[] size;
-	protected final long[] gridPosition;
-	protected final T data;
+	private final int[] size;
+	private final long[] gridPosition;
+	private final T data;
+	private final ToIntFunction<T> numElements;
 
-	public AbstractDataBlock(final int[] size, final long[] gridPosition, final T data) {
+	public AbstractDataBlock(
+			final int[] size,
+			final long[] gridPosition,
+			final T data,
+			final ToIntFunction<T> numElements) {
 
 		this.size = size;
 		this.gridPosition = gridPosition;
 		this.data = data;
+		this.numElements = numElements;
 	}
 
 	@Override
@@ -63,4 +71,10 @@ public T getData() {
 
 		return data;
 	}
-}
+
+	@Override
+	public int getNumElements() {
+
+		return numElements.applyAsInt(data);
+	}
+}

src/main/java/org/janelia/saalfeldlab/n5/BlockReader.java

@@ -25,31 +25,10 @@
  */
 package org.janelia.saalfeldlab.n5;
 
-import java.nio.ByteBuffer;
-
 public class ByteArrayDataBlock extends AbstractDataBlock<byte[]> {
 
 	public ByteArrayDataBlock(final int[] size, final long[] gridPosition, final byte[] data) {
 
-		super(size, gridPosition, data);
-	}
-
-	@Override
-	public ByteBuffer toByteBuffer() {
-
-		return ByteBuffer.wrap(getData());
-	}
-
-	@Override
-	public void readData(final ByteBuffer buffer) {
-
-		if (buffer.array() != getData())
-			buffer.get(getData());
-	}
-
-	@Override
-	public int getNumElements() {
-
-		return data.length;
+		super(size, gridPosition, data, a -> a.length);
 	}
 }

src/main/java/org/janelia/saalfeldlab/n5/BlockWriter.java

@@ -27,14 +27,13 @@
 
 import java.io.IOException;
 import java.io.InputStream;
-import java.io.OutputStream;
-
 import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
 import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
 import org.janelia.saalfeldlab.n5.Compression.CompressionType;
+import org.janelia.saalfeldlab.n5.readdata.ReadData;
 
 @CompressionType("bzip2")
-public class Bzip2Compression implements DefaultBlockReader, DefaultBlockWriter, Compression {
+public class Bzip2Compression implements Compression {
 
 	private static final long serialVersionUID = -4873117458390529118L;
 
@@ -52,36 +51,22 @@ public Bzip2Compression() {
 	}
 
 	@Override
-	public InputStream getInputStream(final InputStream in) throws IOException {
-
-		return new BZip2CompressorInputStream(in);
-	}
-
-	@Override
-	public OutputStream getOutputStream(final OutputStream out) throws IOException {
-
-		return new BZip2CompressorOutputStream(out, blockSize);
-	}
-
-	@Override
-	public Bzip2Compression getReader() {
+	public boolean equals(final Object other) {
 
-		return this;
+		if (other == null || other.getClass() != Bzip2Compression.class)
+			return false;
+		else
+			return blockSize == ((Bzip2Compression)other).blockSize;
 	}
 
 	@Override
-	public Bzip2Compression getWriter() {
+	public ReadData decode(final ReadData readData) throws IOException {
 
-		return this;
+		return ReadData.from(new BZip2CompressorInputStream(readData.inputStream()));
 	}
 
 	@Override
-	public boolean equals(final Object other) {
-
-		if (other == null || other.getClass() != Bzip2Compression.class)
-			return false;
-		else
-			return blockSize == ((Bzip2Compression)other).blockSize;
+	public ReadData encode(final ReadData readData) {
+		return readData.encode(out -> new BZip2CompressorOutputStream(out, blockSize));
 	}
-
 }

src/main/java/org/janelia/saalfeldlab/n5/ByteArrayDataBlock.java

@@ -25,13 +25,14 @@
  */
 package org.janelia.saalfeldlab.n5;
 
+import java.io.IOException;
 import java.io.Serializable;
 import java.lang.annotation.ElementType;
 import java.lang.annotation.Inherited;
 import java.lang.annotation.Retention;
 import java.lang.annotation.RetentionPolicy;
 import java.lang.annotation.Target;
-
+import org.janelia.saalfeldlab.n5.readdata.ReadData;
 import org.scijava.annotations.Indexable;
 
 /**
@@ -49,7 +50,7 @@ public interface Compression extends Serializable {
 	@Inherited
 	@Target(ElementType.TYPE)
 	@Indexable
-	public static @interface CompressionType {
+	@interface CompressionType {
 
 		String value();
 	}
@@ -61,9 +62,9 @@ public interface Compression extends Serializable {
 	@Retention(RetentionPolicy.RUNTIME)
 	@Inherited
 	@Target(ElementType.FIELD)
-	public static @interface CompressionParameter {}
+	@interface CompressionParameter {}
 
-	public default String getType() {
+	default String getType() {
 
 		final CompressionType compressionType = getClass().getAnnotation(CompressionType.class);
 		if (compressionType == null)
@@ -72,7 +73,40 @@ public default String getType() {
 			return compressionType.value();
 	}
 
-	public BlockReader getReader();
+	// --------------------------------------------------
+	//
+
+	/**
+	 * Decode the given {@code readData}.
+	 * <p>
+	 * The returned decoded {@code ReadData} reports {@link ReadData#length()
+	 * length()}{@code == decodedLength}. Decoding may be lazy or eager,
+	 * depending on the {@code BytesCodec} implementation.
+	 *
+	 * @param readData
+	 * 		data to decode
+	 *
+	 * @return decoded ReadData
+	 *
+	 * @throws IOException
+	 * 		if any I/O error occurs
+	 */
+	ReadData decode(ReadData readData) throws IOException;
+
+	/**
+	 * Encode the given {@code readData}.
+	 * <p>
+	 * Encoding may be lazy or eager, depending on the {@code BytesCodec}
+	 * implementation.
+	 *
+	 * @param readData
+	 * 		data to encode
+	 *
+	 * @return encoded ReadData
+	 *
+	 * @throws IOException
+	 * 		if any I/O error occurs
+	 */
+	ReadData encode(ReadData readData) throws IOException;
 
-	public BlockWriter getWriter();
-}
+}