Splitting ZipArchiveEntry content

[exelsior]

Hello.
I got problem with splitting content of ZipArchiveEntry while sending it to other service.
Context: i have service with zip creating function. I got some data from third party service, generated csv file based on this data and put it into zip archive. After that, i got memorystream (which is parent for zip), read it till the end, got all contents from there, send it to another service and clean mother-stream. In my case this is necessary because there is huge possibility that data from third party services will be huge, so i need to save opportunity for chunk sending. After closing zip with using, i send eocd bytes. But when i'm trying to recreate zip archive from those bytes, i got an exception that number of entries in CD doesn't corresponding with the actual number.

And here is my question. Is it possible to split ZipArchiveEntry content while sending it or processing it? Sole purpose of this action is reduce memory consuming.
I wrote sample. There is no sending or other stuff. It is just converting those bytes to Base64 there and back.

using System.IO.Compression;
using System.Text;
using CsvHelper;
using CsvHelper.Configuration;

internal class Program
{
    public static readonly List<byte> Bytes = new List<byte>();
    public static void Main(string[] args)
    {
        using (var memoryStream = new MemoryStream())
        {
            using (var zipArchive = new ZipArchive(memoryStream, ZipArchiveMode.Create, leaveOpen: true))
            {
                var entry = zipArchive.CreateEntry("entry.csv");
                using (var writer = new StreamWriter(entry.Open(), Encoding.UTF8, leaveOpen:true))
                {
                    var csvConfiguration = new CsvConfiguration(CultureInfo.InvariantCulture)
                    {
                        Delimiter = ";"
                    };
                    var list = new List<string>
                    {
                        "Val1", "Val2", "Val3"
                    };
                    var list1 = new List<string>
                    {
                        "Val4", "Val5", "Val6"
                    };
                    using (var csv = new CsvWriter(writer, csvConfiguration, leaveOpen:true))
                    {
                        Write(memoryStream, writer, csv, list);
                        Write(memoryStream, writer, csv, list1);
                    }
                }
            }
            Bytes.AddRange(Convert.FromBase64String(Convert.ToBase64String(memoryStream.GetBuffer().AsSpan(0, (int)memoryStream.Length))));
        }

        var bytes = Bytes.ToArray();
        
        using var mStream = new MemoryStream(bytes);
        var arch = new ZipArchive(mStream);
        var entries = arch.Entries.First();
    }
    
    public static void Write(MemoryStream stream, StreamWriter writer, CsvWriter csv, List<string> values)
    {
        foreach (var val in values)
        {
            csv.WriteField(val);
        }

        csv.NextRecord();
        csv.Flush();
        writer.Flush();
        Bytes.AddRange(Convert.FromBase64String(Convert.ToBase64String(stream.GetBuffer().AsSpan(0, (int)stream.Length))));
        stream.Clear();
    }
}

public static class MemoryStreamExtensions
{
    public static void Clear(this MemoryStream source)
    {
        var buffer = source.GetBuffer();
        Array.Clear(buffer, 0, buffer.Length);
        source.Position = 0;
        source.SetLength(0);
    }
}```

[huoyaoyuan]

ZipArchive should be using the state of underlying stream. I don't have enough knowledge about the zip file format.
You may consider writing directly to DeflateStream which wraps a network stream etc.

@dotnet/area-system-io-compression for better insights.

[bartonjs]

There's something wrong in your code to extract data out of the MemoryStream. Running your code as-is reproduces the System.IO.InvalidDataException: Number of entries expected in End Of Central Directory does not correspond to number of entries in Central Directory. error, but running code with an "equivalent" custom Stream implementation doesn't. I recommend just writing a custom Stream type over trying to work out where your interfering with MemoryStream is going wrong.

private static void Main()
{
    List<byte> bytes = new();

    using (Stream stream = new ListWritingStream(bytes))
    using (ZipArchive archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true))
    {
        ZipArchiveEntry entry = archive.CreateEntry("entry.csv");

        using (StreamWriter writer = new StreamWriter(entry.Open(), Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("Val1;Val2;Val3");
            writer.WriteLine("Val4;Val5;Val6");
        }
    }

    using (Stream stream = new MemoryStream(bytes.ToArray()))
    using (ZipArchive archive = new ZipArchive(stream))
    {
        Span<byte> buf = stackalloc byte[1024];

        foreach (ZipArchiveEntry entry in archive.Entries)
        {
            Console.WriteLine($"{entry.FullName} -- {entry.Length} byte(s)");

            if (entry.Length <= buf.Length)
            {
                int read;

                using (Stream entryStream = entry.Open())
                {
                    read = entryStream.ReadAtLeast(buf, (int)entry.Length);
                }

                Console.WriteLine("---");
                Console.WriteLine(Encoding.UTF8.GetString(buf.Slice(0, read)));
                Console.WriteLine("---");
            }
        }
    }
}

private class ListWritingStream : Stream
{
    private readonly List<byte> _list;

    public ListWritingStream(List<byte> list)
    {
        _list = list;
    }

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        throw new InvalidOperationException();
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        throw new InvalidOperationException();
    }

    public override void SetLength(long value)
    {
        throw new InvalidOperationException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        for (int i = 0; i < count; i++)
        {
            _list.Add(buffer[checked(offset + i)]);
        }
    }

    public override void Write(ReadOnlySpan<byte> buffer)
    {
        foreach (byte b in buffer)
        {
            _list.Add(b);
        }
    }

    public override bool CanRead => false;
    public override bool CanSeek => false;
    public override bool CanWrite => true;
    public override long Length => throw new InvalidOperationException();

    public override long Position
    {
        get => throw new InvalidOperationException();
        set => throw new InvalidOperationException();
    }
}