ZLibStream discards rest of input buffer after inflation

[wegylexy]

Even when using a PipeReader, a ZLibStream reads more than it needs to inflate up to the required number of bytes (uncompressed size). How can I read the unprocessed bytes?

Use case: parsing git packs without indices. A git pack consists of PACK|version|entries|header|zlib|header|zlib|header|zlib|... where every header includes the uncompressed size but not the deflated size. Native zlib can tell the number of consumed bytes such that the caller can resume its own processing without losing unused buffered bytes.

A PipeReader contract for ZLib is preferred over Stream.

The following hack using LambdaExpression.Compile() may work with AOT.

class ZLibStreamInputBufferExtension
{
    private static readonly Func<ZLibStream, ReadOnlyMemory<byte>> _getInputBuffer;

    static ZLibStreamInputBufferExtension()
    {
        var stream = Expression.Parameter(typeof(ZLibStream));
        var _deflateStream = Expression.Field(stream, "_deflateStream");
        var _zlibStream = Expression.Field(Expression.Field(_deflateStream, "_inflater"), "_zlibStream");
        var _buffer = (Func<ZLibStream, byte[]>)Expression.Lambda(Expression.Field(_deflateStream, "_buffer"), stream).Compile();
        var AvailIn = (Func<ZLibStream, uint>)Expression.Lambda(Expression.Property(_zlibStream, "AvailIn"), stream).Compile();
        var NextIn = (Func<ZLibStream, nint>)Expression.Lambda(Expression.Property(_zlibStream, "NextIn"), stream).Compile();
        _getInputBuffer = stream =>
            (int)AvailIn(stream) is not 0 and var availIn && _buffer(stream) is not null and var buffer ?
            new(buffer, (int)(NextIn(stream) - Marshal.UnsafeAddrOfPinnedArrayElement(buffer, 0)), availIn) :
            default;
    }

    public static ReadOnlyMemory<byte> GetInputBuffer(this ZLibStream stream) =>
        _getInputBuffer(stream);
}

[rhuijben]

See also #62113

[AraHaan]

As much as I hate to say this, that implementation might break when any of those private symbols gets renamed in the codebase.

I think what you are looking for is a span based implementation like the encoder/decoder apis in my proposal.

Infact, I would love to ship an external library version of that implementation in case it does not make it entirely into .NET 7 however it will be tricky to have it load a modified version of System.IO.Compression.Native because my change required adding an additional symbol to it, it would also rely on internal bits of System.IO.Compression as well too to work.