Restructuring cargo project using workspaces

[WolverinDEV]

It becomes more and more clear, that conserve currently consists of two parts: the library and the cli interface
Therefore it's more sensible to split up these two parts. This will be especially obvious with PR #173
as the cli interface no longer consists of one single .rs file and the library itself gets more universally applicable.

Thus I suggest, separating the cli interface into a separate crate (producing a binary file) by utilizing cargo workspaces.
This also allow, writing more specified tests as well as benchmarks specific for the library / the cli interface.

Attention:
Restructuring should be considered after merging #173.
Therefore #173 blocks this.

[sourcefrog]

I have mixed feelings about doing this right now: it seems too soon.

On the one hand, I agree, there is some core functionality and then a terminal/CLI interface to it. Keeping the layering clean, as in #173, is good for code clarity and testability even when it ships as a unitary crate.

On the other hand: It's already possible to write separate tests for the binary vs the API, and in fact there already are separate tests for them. It doesn't seem like splitting into multiple crates would really help with developing or testing the CLI in any identifiable way? I think it's also possible to have out-of-tree binaries depend on the conserve library API even though the conserve crate also provides binaries, if you want to do that for experiments or benchmarking.

The main effect of this would seem to be making the library API a separately publishable crate and kind of more prominent. But, I think of the library API today as making no stability guarantees, having only one client (the CLI), and subject to big changes to for example introduce monitors or whatever. Obviously the library can be made "more public" without promising stability, but, why?

Also, if the library is more prominent as a crate in itself, then it would seem to need to be tested more thoroughly, and, I'm not so sure that's worthwhile if it only has one client. In general I'm tilting more these days towards primarily testing the public interface and specifically for Conserve towards saying that today, the CLI is the public interface. In particular, for this kind of tool, I think having a CLI that is well scriptable (e.g. with json output) is probably more often useful than having an unstable Rust API.

So, that's how I see it; I'm -0.5 on splitting it right now. But separating terminal/CLI UI from other things inside one crate is welcome.

[WolverinDEV]

I understand your point of view.
It's reasonable to say, that if the CLI is the only public interface.
Why bothering to separate them in the first place?

On the other hand, I see conserve as a specification on a backup format aiming towards high performance and liability including features
as compression, deduplication, and (asymmetric) encryption. Therefore the library would be the implementation of
the specification and the CLI one possible tool to use conserve.

Yes, making the library more public without promising stability isn't that reasonable right now.
But it will be the first step of having a public library. Guaranteeing stability can be done as second step
especially due to some pending ground braking API changes it's currently not yet possible.

The whole idea is to separating the library from any possible user interface
as early as possible so intuitively nudge any developer into doing so as well (separating conserve specification implementation from the ui implementation).
Taking the step, to split theses parts up is a stronger approach on separating terminal/CLI UI from other things inside one crate.

[sourcefrog]

At the moment I'm just more interested in (in no particular order)

• improving the format, which has several aspects including encryption
• a clean separation of different concerns inside one single tree, including the UI/CLI from other things, but that is not the only layer that should be clean
• having strong validation checks for the format that can catch every problem
• having strong test coverage at some level, primarily on the CLI -- ideally so that every mutant from cargo-mutants is caught
• generally cleaning up the code, which began as the first significant thing I wrote in Rust
• automatically expiring old backups
• automatically expiring enough backups to make space for a new backup on fixed-size storage
• supporting cloud storage

I think nudges or refactorings towards a separation of UI from core can be done equally well and with less friction in a single crate.

To make the UI separation stronger I suspect it might be better to add a different UI inside the same crate, or in subcrates that use the conserve interface. e.g. conserve could have a fullscreen/curses UI for backups, or a web UI, or a headless FUSE read interface.

I actually don't really want anyone using the library interface externally, at least until it's more mature.

[sourcefrog]

I was thinking about this a bit more the other day, particularly with the appearance of a FUSE server as a library client.

A clean library interface that can be used to build other UIs or tools is a goal for this project: not the top goal for me at the moment but a goal.

I think the library interface is currently too imperfect to make any promises about stability. I would like it to get there, but much of it still reflects that I was learning Rust or thinking about how to structure the problem.

Along the way, it's worth being clear what belongs in the CLI and what in the library, and building that in two layers. There is also some "terminal UI" stuff, like progress bars, that's not exactly locked to the CLI and could be used by other terminal clients, but also not exactly part of the core library...

As far as making them actually separate crates: I guess the only cost is making the split, and then some slight ongoing cost that they each need to be released and versioned separately.

[WolverinDEV]

Hey,
nice to head back from you on such an old topic.
I like how you abstracted my idea on a monitor and made conserve more modular.

In respect to splitting up into different crates, I highly welcome that change.
In regards to the library interface, I would argue that in order to get things started, there is no direct need to introduce API stability. Over time such stability can be introduced, as third parties actually start to depend on it.

A hard but pretty handy to have feature would be to incorporate no_std support for the core crate.
Thus would allow to create a Windows driver as well.

Edit:
PS: It's good to see shifting your focus more into conserve again 👍

[sourcefrog]

Tell me more about the Windows driver idea? (Maybe in a discussion thread, to keep this issue focused.)

I wonder if it would be easier or not to have a no_std driver that communicates with a userspace service that can use std... Writing the core in no_std sounds daunting, but then I've never written any. In fact I was thinking about perhaps changing it to Tokio, which would put that even further away.