Why is S3 config option 'stream_reads' false by default?

[adi64]

Hi there 👋
In our application, we handle large files. Typically the raw data is in S3 storage. We copy files from S3 to local storage, process them, and move the results to S3 for distribution. The files that we process usually are larger than main memory. To copy files, we use readStream and writeStream.
Now I noticed that this works perfectly fine when copying files from local storage to S3. However, copying from S3 to local storage causes a problem: The /tmp folder of the system (which often is on a ram-disk!) would grow huge with temporary data.
It seems that readStream on an S3 disk actually downloads the whole file into a temporary location and then gives me a stream resource to that temporary file, instead of letting me stream from S3 directly.

After digging for a bit, I found out that Laravel disables streamed reads by default, unless the undocumented (!) stream_reads option of the S3 disk config is set:

framework/src/Illuminate/Filesystem/FilesystemManager.php

Line 244 in 37cb345

$streamReads = $s3Config['stream_reads'] ?? false;

IMHO this is pretty unexpected, because I know Laravel mainly wraps Flysystem, and that streams by default. This is also what I would expect from a method that's called readStream!

An attempt was made to add that to the docs in laravel/docs#8371, but it was closed because it was missing details. I'd like to take another stab at this one, but to do so, I think I need to answer a question:
Why are streamed reads disabled by default for S3 disks?

@driesvints, maybe you are reading this. It seems like you added that logic in #34001, could you shed some light on this?

[driesvints]

I don't know anymore why it was false by default, sorry. Most likely because it was a new feature at the time.

[adi64]

Thanks for the really quick reply!
Do you think the default could be changed? Not sure if changing the default would be considered a breaking change in behavior.
Anyways, I'll think of a way to add this to the docs :)

[driesvints]

I really don't know sorry. It's been so long I touched this piece of the framework.

[bryanbeginly]

This was killing me for two days. The file was streaming from the public disk no problem, but I couldn't get it to load from s3 for anything. Finally, after some googling, I found this. Thanks for figuring it out @adi64 this really unblocked me.

[bmckay959]

This also bit me. Thanks for this.

I'm using the league CSV which requires the stream to be seekable, but that doesn't appear to work with the Storage facade for S3 for some reason. Did any of you have to deal with seekable streams and can point me in a good direction?

[adi64]

Not really sure on this since I don't need seekable streams. I think if you disable "proper" streamed reads (i.e. set 'stream_reads' => false in your storage config for the S3 disk), then it will download the file to a temporary location and give you a resource that is basically a local file handle, and therefore I think you could maybe seek in there.
But, again, not sure if that really works, and additionally this has the problem of internally downloading the entire file beforehand.
I don't think the League S3 adapter offers a truly streaming interface that also supports seeking. Maybe you'd have to find another implementation or write one yourself that uses the Range: header behind the scenes.