What is the design behind System.IO.Pipelines

[HurricanKai]

Stream is a layered API, if I want apply some pre/post processing to a stream, I just wrap the existing stream in a stream that does the operations I want.
What is the idea here with System.IO.Pipelines? Should I do the same, and chain pipes to do my work?
Or should I instead do manual buffering and use only a single pipe to read data from, then do my steps on the read data?
It's certainly possible to create highly efficient pre/post processing pipes, but I don't think I've ever seen one, but I don't think I've seen any advanced use cases at all...
Looking at the core types, it seems I should do things manually, or types are missing? Or should I make these pipes as I need them?

[davidfowl]

What is the idea here with System.IO.Pipelines? Should I do the same, and chain pipes to do my work?

You should do the same. Decorating an existing PipeReader/PipeWriter is the most efficient way to expose these pipelines.

Or should I instead do manual buffering and use only a single pipe to read data from, then do my steps on the read data?

This is also possible and easier but less efficient. Ideally you would only introduce your own buffer when the user decides to buffer their own data.

It's certainly possible to create highly efficient pre/post processing pipes, but I don't think I've ever seen one, but I don't think I've seen any advanced use cases at all...

There's a higher level API here I've been experimenting with https://github.com/davidfowl/BedrockFramework/tree/main/src/Bedrock.Framework/Protocols.

There are also lots of decorator PipeReader/PipeWriter in http://github.com/dotnet/aspnetcore/

Looking at the core types, it seems I should do things manually, or types are missing? Or should I make these pipes as I need them?

Most of the core types are optimized for wrapping a Stream and adding some buffering logic on top. This is why we haven't shipped any other native PipeReader/PipeWriter out of the box.

[HurricanKai]

Thank you, if I'm understanding I'm supposed to do:

class MyPipeWriter : PipeWriter
    {
        private readonly PipeWriter _underlying;
        private IMemoryOwner<byte>? _current;

        public MyPipeWriter(PipeWriter underlying)
        {
            _underlying = underlying;
        }
        
        public override void Advance(int bytes)
        {
            var actualMemory = _current.Memory.Slice(0, bytes);
            DoActualAction(actualMemory);
            _current = null;
        }

        public override Memory<byte> GetMemory(int sizeHint = 0)
        {
            if (_current is not null)
                ThrowInvalidOperation();
            if (sizeHint <= 0)
                sizeHint = -1;
            
            _current = _pool.Rent(sizeHint);
            return _current.Memory;
        }

        public override Span<byte> GetSpan(int sizeHint = 0)
        {
            return GetMemory(sizeHint).Span;
        }

        public override void CancelPendingFlush()
        {
            _underlying.CancelPendingFlush();
        }

        public override void Complete(Exception? exception = null)
        {
            _underlying.Complete(exception);
        }

        public override ValueTask<FlushResult> FlushAsync(CancellationToken cancellationToken = new CancellationToken())
        {
            return _underlying.FlushAsync(cancellationToken);
        }
    }

And in "DoActualAction" I have the user-provided data and can work with the underlying pipe to write the actual data.
Seems like this will incur a lot of buffering if done multiple times? I guess the MemoryPool would be sort of fine due to this often using the default size, and a lot of sharing being possible, but still a lot of memory pressure.

[davidfowl]

It could happen in Advance like you have it, or it can happen in FlushAsync. If you do it in FlushAsync then you'll need to buffer more than a single segment, but the idea is the same. This buffering logic could be encapsulated in a helper.

PS: Your logic is flawed in that, you need to write your buffer to the underlying PipeWriter somewhere.