Exception when reading partial shard that only contains fill value pixels

[melissalinkert]

Running this v3 test code:

import java.util.Arrays;

import dev.zarr.zarrjava.store.FilesystemStore;
import dev.zarr.zarrjava.v3.Array;
import dev.zarr.zarrjava.v3.ArrayMetadata;
import dev.zarr.zarrjava.v3.DataType;

public class V3Test {

  public void writeTestZarr(String outputPath) throws Exception {
    FilesystemStore store = new FilesystemStore(outputPath);
    int arraySize = 17;
    int shardSize = 8;
    Array data = Array.create(store.resolve("0/0"),
      Array.metadataBuilder()
        .withShape(arraySize, arraySize)
        .withDataType(DataType.UINT8)
        .withChunkShape(shardSize, shardSize)
        .withFillValue(0)
        .withCodecs(c -> c.withSharding(new int[] {shardSize, shardSize}))
        .build()
    );

    for (int y=0; y<arraySize; y+=shardSize) {
      int height = (int) Math.min(shardSize, arraySize - y);
      for (int x=0; x<arraySize; x+=shardSize) {
        int width = (int) Math.min(shardSize, arraySize - x);
        byte[] tile = new byte[width * height];
        Arrays.fill(tile, (byte) 0);

        int[] shape = new int[] {height, width};
        final ucar.ma2.Array pixels = ucar.ma2.Array.factory(
          data.metadata.dataType.getMA2DataType(), shape, tile);
        data.write(new long[] {y, x}, pixels);
      }
    }

    data.read(new long[] {arraySize - 1, arraySize - 1}, new int[] {1, 1});
  }

  public static void main(String[] args) throws Exception {
    new V3Test().writeTestZarr(args[0]);
  }
}

successfully creates and writes an array, but throws this exception when attempting to read the bottom-right-most pixel:

Exception in thread "main" java.lang.RuntimeException: dev.zarr.zarrjava.ZarrException: Could not read shard index.
	at dev.zarr.zarrjava.v3.Array.lambda$read$0(Array.java:185)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at dev.zarr.zarrjava.v3.Array.read(Array.java:154)
	at com.glencoesoftware.V3Test.writeTestZarr(V3Test.java:40)
	at com.glencoesoftware.V3Test.main(V3Test.java:44)
Caused by: dev.zarr.zarrjava.ZarrException: Could not read shard index.
	at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodeInternal(ShardingIndexedCodec.java:198)
	at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodePartial(ShardingIndexedCodec.java:250)
	at dev.zarr.zarrjava.v3.codec.CodecPipeline.decodePartial(CodecPipeline.java:93)
	at dev.zarr.zarrjava.v3.Array.lambda$read$0(Array.java:173)
	... 5 more

Changing just the fill value upon array creation or the pixel values being written results in the test code running without error.

An initial investigation suggests that this method:

https://github.com/zarr-developers/zarr-java/blob/main/src/main/java/dev/zarr/zarrjava/store/FilesystemStore.java#L54

assumes the relevant chunk/shard file always exists, but that isn't the case if all array values happen to be the fill value. I don't know if FilesystemStore is the correct place to fix it, though. If this is actually expected behavior, then any suggestions on how to prevent this when writing/reading arrays would be appreciated.

A similar error was reported when converting LuCa-7color_Scan1.qptiff, in glencoesoftware/bioformats2raw#290 (comment). This data has large areas of fill-value-only pixels around the perimeter of the image, in particular along the right-hand side.