Centralized Key storage concerns

[joehand]

There are a few impacts of storing secret key in one place:

1. If data storage is not centralized, you could have two applications writing to the same dat stored in two separate locations.
2. Two separate applications could write to same dat in same place (a lock of some sort would solve this).
3. A user that wants to download their own Dat as a non-writer couldn't (e.g. Dat would look up the secret key for that Dat, find it, and make it writable).

[joehand]

IRC discussion:

pfrazee> yoshuawuyts: are dat-cli and dat-desktop going to unify their storage for all dat internal data (eg under ~/.dat) or only unify the key storage?
9:52 AM <•yoshuawuyts> pfrazee: think the latter, jhand was working on splitting the keys off from the dirs not too long ago
9:52 AM <pfrazee> yoshuawuyts: ok cool
9:52 AM <•yoshuawuyts> pfrazee: we might like agree on common locations and stuff
9:52 AM <•yoshuawuyts> pfrazee: and need to figure out a way to make sure we can lock a resource
9:52 AM <•yoshuawuyts> pfrazee: the latter i can imagine is relevant for beaker too haha
9:53 AM <pfrazee> yoshuawuyts: that would be really great, yeah
9:53 AM <pfrazee> yoshuawuyts: I'm going to unify key storage for sure
9:53 AM <pfrazee> yoshuawuyts: if we unified data storage too, then I could have dat-cli running for something I'm working on, and make edits to it from within beaker
9:54 AM <•yoshuawuyts> pfrazee: we want to allow people to share random dirs tho - like with git
9:55 AM <pfrazee> yoshuawuyts: might still work even if you dont centralize data storage, though we'll have to duplicate the internal data structures
9:55 AM <pfrazee> yoshuawuyts: the real challenge is making sure two active writers work together nicely
9:56 AM <jhand> pfrazee: Im against centralizing the data storage
9:56 AM <•yoshuawuyts> pfrazee: haha, yeah that's tricky - the only way I can imagine that working well is if there's only 1 writer at the same time
9:56 AM <jhand> pfrazee: but the storage for a given dir can still be used across apps
9:56 AM <pfrazee> yoshuawuyts: we might need to have an archive-wide lock to accomplish that
9:57 AM <pfrazee> jhand: how would that work? Beaker needs to know where the dir is
9:57 AM — •yoshuawuyts offline, going to a meetup
9:57 AM <jhand> pfrazee: ya you need some db for where dirs are stored 
9:57 AM <pfrazee> jhand: why are you against centralizing the data storage?
9:57 AM <•karissa> pfrazee: you could use multidat
9:58 AM <•karissa> pfrazee: copying large datasets around is one problem i can think of
9:58 AM <jhand> pfrazee: for CLI at least, I think it causes more problems than it'd be worth
9:58 AM <jhand> it obfuscates a lot of what is going on
9:59 AM <jhand> or just makes you do the dropbox/gdrive type thing
9:59 AM <pfrazee> jhand: I do think it's an advantage to be able to delete the folder and have the history disappear with it
10:00 AM <•karissa> it would cause issues with someone who has a dat on an external drive
10:00 AM <pfrazee> karissa: true
10:00 AM <•karissa> for big data we need to bring the dat to the data rather than the data to the dat
10:00 AM <jhand> pfrazee: there is actually a lot of discussion in IPFS repos about benfits/issues with centralized storage
10:01 AM <pfrazee> karissa: jhand: the only significant cost of not centralizing the data storage is, like with beaker, any time you have multiple programs using it, they'll each have to duplicate the internal structures (aka the history)
10:01 AM <•karissa> pfrazee: if they use the same conventions they dont
10:01 AM <pfrazee> karissa: beaker has to store its dat data centrally
10:01 AM <jhand> pfrazee: ya whats wrong with storing history in place and using same storage convention?
10:01 AM <•karissa> this is why i also like the idea of restricting dats to being folders (liek git) and self-contained
10:02 AM <pfrazee> if I open some dat in beaker, it has no concept of where that dat might be on the local FS
10:02 AM <pfrazee> it receives the dat via network interfaces provided by dat cli
10:02 AM <jhand> ah right
10:02 AM <•karissa> i tihnk this is what multidat attempts to fix
10:02 AM <jhand> I was actually thinking about that last night
10:03 AM <pfrazee> karissa: actually I think multidat lets you access more than one distinct dat
10:03 AM <•karissa> pfrazee: i belive it keeps a centralized db that relates key to path
10:03 AM <jhand> pfrazee: we could store the path in the ~/.dat folder too
10:03 AM <•karissa> pfrazee: hmm
10:03 AM <jhand> oh ya does mutlidat do that with toilet
10:04 AM <pfrazee> jhand: karissa: mmm I have pretty mixed feelings about that. I'd have some dats in random locations, some of them central. I think users might be surprised if they were connected
10:04 AM <pfrazee> wouldnt you expect beaker to have its own data cache?
10:05 AM <jhand> pfrazee: back in a bit we're having a team call
10:05 AM <pfrazee> ok
10:05 AM <pfrazee> just to tail this off, what's at stake here is simply duplication of data, which I'm not overly concerned with
10:05 AM <pfrazee> it's inefficient but not a big deal. What *is* a big deal is coordinating around multiple active writers for a dat
10:06 AM <pfrazee> so we'll need to figure out how to set things up so that the CLI can actively host a dat, then have beaker nav to the dat and write to it