make disk blobstore multilevel

[Gargaj]

Currently when using a local blobstore, all blobs (per account) are just dumped in a single blocks dir, which is going to slow down disk access over time once an account hits over, say, 10k blobs. (Not unreasonable to expect.) By contrast, the actors dir has at least a single level of distribution into 256 subfolders. Any chance to have the same kind of distribution for blocks?

[axfelix]

To be honest, this is not an area of focus because when you're going to production, you should be using Object storage for blobs rather than keeping them in the directory. We should probably look at making an interactive setup script that configures an S3 endpoint eventually.

[Gargaj]

That assumes you're going into "production" and not just hosting your stuff in a low capacity cupboard server (i.e. the "P" in "PDS")

[axfelix]

I would still suggest provisioning your own object storage service (Minio or something like it) on the cupboard infrastructure in that case — there’s lots of solutions for DIY S3, all of which will scale better than trying to implement more complicated filesystem storage. But I’d like us to make this clearer!

[Gargaj]

So if I already use disk blobstore, then surely I would need to take care of the migration of existing data into a DIY S3...?

Sounds like something that really needs to be highlighted during/before installation, or just remove disk-based entirely.

[mcprostar205]

Requiring object storage defeats the purpose of being able to self-host. OS directory limits (and inode depletion) are well-known and have been addressed by software for decades.

Why place such an artificial restriction the requires cloud provider hosting when a simple technical solution creating sharded folder structures exists?

[Gargaj]

FWIW I just migrated my own to a locally hosted MinIO and, hilariously, it has literally the same problem: each object is just stored as a directory in a root, so the 1800 blobs are now 1800 directories, so that was basically a waste of time and computing, and I might as well revert it back.

[benvanstaveren]

I don't agree with the "set up your own S3 in the cupboard" sentiment; at work we run serious production things on serious hardware, where S3 isn't an option, so it's disk based storage. In production :)

And it's not like it's all that complicated IMO to add nesting; take the last 3 characters from the CID, split, join /, and there you go, your nested path part. Same for getBlob, could just try to get it from the current location, and if not found, do the nested thing, or vice versa whichever would guarantee more first-try-success.

🤷