Oxidize GitButler

[Byron]

Used Git2 Features

The list is probably not completely exhaustive, omitting smaller features intentionally as they are covered, and focussing on the bigger picture.

• painless changes to HEAD, through a specific abstraction.
• Git 3 support
  • ref-table
  • SHA256
• wanted for basic head-info gitbutlerapp/gitbutler#8451
  • A quick way to go from a ref-name to a ref and id. Maybe provide a connected ref-name type to make that easy and general?
  • compare &FullName to &FullNameRef
• Should FullNameRef::shorten() return a PartialName because it can?
  • Probably it's better to document other ways to get a partial name from a full name, and why one probably doesn't want to store it in structures.
• RelativePathBuf type, which could be used well in merge related types, including conflicts.
• make it convenient to obtain hashes from worktree content. diff_resource_cache() can be used, but it's a bit indirect.
• anything with relative paths is annoying to use if it's Path sometimes even though the user has a BStr - make it use ToComponents instead and make it broadly available.
• make it easy to get the local tracking branch from a remote tracking branch, taking branch.<name>.merge into consideration
• replace ignore crate with gix-dirwalk
• a way to create a local tracking branch for a remote tracking branch (with all the setup).
• status for embedded repos (i.e. local-only repositories outside of .gitmodules) for Git compatibility
• an easy way to get a path from a repo + BString relapath
• graph-traversal with ancestor exclusion similar to hide()
• refspecs
• init (and fail if present already) , bare or non-bare
• get configuration by scope
• set configuration by scope and write it back while keeping comments/nonedestructively
  • writes are convenient through high-level API
  • easy acquisition of the latest configuration.
• read and traverse git configuration
• fetch a remote (and handle credentials like Git would)
• merge-base octopus to prevent surprising behaviour - make sure this is made available in virtual merge base handling as well.
• full support for git-config, includeIf and hasconfig
• merge multiple commits into one, with multi-mergebase support (see checkout_branches…())
  • provide merged index
  • merge submodules
• tree-from-worktree, but fast using full status
• make it easy to load all relevant Git configuration, on Windows particularly - maybe go for correctness first…
• something like blob_writer() with auto-conversion with filters if a hint-path is given.
• write loose objects uncompressed, see the issue around big files
• index.add() for entries, conflicted entries handling, and deletion of entries as well, ideally efficiently.
  • a nice way to edit index entries, but needs to support lookups at the same time.
• in-memory .gitignore rules, so that any user of excludes gets these automatically
• sanitization as part of refname validation (just like Git offers, see this issue)
• add ODB alternates
• add submodule
• use git-remote helpers, i.e. codecommit or rad.
• 'is descendant of' query
• ahead/behind information - no need anymore for V3 as it calculates this as graph byproduct
• merge base query
• create a new remote (primitives exist, but there is no higher-level method for it)
• rename reference (there is no shortcut for this, just lower-level primitives delete+create)
• git add . - with proper worktree filtering (add currently unsupported, but worktree filter could help already)
• diff tree to workdir (primitives exist)
• merge trees
• rebase (used in one test-function only)
• commit() - new commit without setting the reference
• gix-testsupport supports parallel writable tests
• find remotes
• find head and deal with refs in general (peeling, etc)
• diff tree with tree
• find objects and deal with them in various ways
• revwalk (even though it's still to be seen if the algorithm is what's expected, see the topo-walk discussion
• query excludes/.gitignore - but should be performed on all available paths
• get the index
• cherry pick
  • implemented with merge and in a custom form. It's also not a lot of code.
• create a new commit
• create a new signed commit (and respect these settings even when rebasing)
• build a new tree by API
• obtain remote branch names in various flavours
• git status (primitives exist, WIP)
• unconditional reset of specified paths in a tree (or maybe even the diff between trees
  • Needs 'keep going' and submodule support
• optionally run hooks on commit (and other events maybe, just like Git)
  • keep in mind what GUI applications would do, and maybe document how to solve the issue that the env isn't the same as in the terminal. See Use default shell for git-hooks instead of bash gitui-org/gitui#2317 .
• walk a tree
• index to tree
• tree to index
• complete diff based on git status equivalent (primitives exist) - needs diff of tree to index.
• support for merge filters to further reduce the chance of conflicts
• gix should handle renames like a -> a/new - but currently needs special handling
  • This will natively work with ref-table and in Git3.0

Wanted Features for GitButler

• be a stock Git client without the need for a remote
  • Special features can be put on top, but they always need to interact well with dropping down to stock Git.
• make it work without the need for an integration commit, make the workspace with 'horizontal' integration optional
• use real database for its own data
• make hunk-dependency computation flawless
• Support for Tags and other Git primitives, without specialty
• all managed data/commits/branches can deal with manual user intervention
• first-class multi-window support - so useful for multiple/big screen setups
  • Also support spinning of views, like diffs of commits, like in Fork
• one way for meta-data - avoid dumping GitButler specific data into commit headers unless they are valuable permanently. Everything else should go into its own store.
• "The right abstractions" to assure the frontend can't end up in an invalid date

Correctness Improvements

• use gitoxide for all configuration access - easy and makes per-user/per-project configuration work right away
• commits are created by manipulating the index first, to make hooks work properly which look at the staging area.
• handle worktree filters
• deal with GPG and ASKPASS password prompt programs that don't work without parent shell
• run all hooks and do so correctly
• allow committing submodules
• awareness of Git states, like 'rebasing', 'cheery-picking' - gix-sequencer can do that.
• pick up login shell environment variables - to let invoked programs behave correctly, like signing. Maybe configurable.
• respect Git configuration by default - i.e. sign commits if enabled without relying on GitButler specific variable.
• Atomic operations in the backend (either everything is changed, or nothing)

Wanted

What follows is prominent issues from the GitButler tracker.

• Azure DevOps cloning support.
• Windows Support
  • natively supported
  • there may be rough edges around program and hook invocations, to be smoothed out of time (probably this is about more than Git though, currently unclear)
• SSH Authentication
  • ssh is used, which is exactly what git does.
  • on Windows, there is no native ssh transport yet which might be worse than git2 right now. See Built-in SSH transport #1246 for more.
• Git LFS support
  • checkouts will call filter programs, thus checkout LFS files
  • diffs will apply all diff-related features and handle LFS files correctly
  • additions, i.e. git add, aren't possible yet, but when they are they will call worktree filters
• Opening Submodules
  • Works in theory, even though it's unclear what's happening that it's not working currently
• Local ref handling
  • Making these entirely transparent to git log --all might be possible with namespaces. Maybe worth a try.
  • This is feasible once gitoxide is used everywhere - each repository instance created needs to have the GitButler namespace set then, at least those that read or write GitButler references.
  • However, using these would probably yield no benefit over how it's done currently.

Here is the repo: https://github.com/gitbutlerapp/gitbutler

Shortcomings that might be a problem

• copy detection differs from git weirdly #1524

Guiding Principles

After our in-person meet, I kept mentioning a few driving principles that I find valuable to keep in mind when planning the next steps.

• The transition to gitoxide should be step-wise.
• Each step should yield meaningful, perceivable value for the user of the application.
• It's OK to not reach goals directly, i.e. use intermediate mechanisms that help now, knowing that they will be removed later.

gitoxide is an implementation detail, but ultimately it will drive down the implementation complexity of the application while improving its performance and compatibility.

Notes on issues

From time to time, one can look into issues that are waiting for feedback.

Try to fix this when converting to gitoxide

Check the `gitoxide label for more.

Big Problems

Architecturally, there might be issues in resolving these:

• Issues applying a branch with conflicts gitbutlerapp/gitbutler#3319 (resolving conflicts with the usual set of tools)

Requested features

Roughly in order of effort-for-value

• Open gitbutler in a repository from the terminal with gitbutler . (see this comment)
• Better handling of being offline (see this issue as example)

Authentication

Use this label to list them all.

Profile Picutures

• Profile picture doesn't get updated at all places gitbutlerapp/gitbutler#2589
• Adding the Linux Kernel as project takes more than an hour to complete gitbutlerapp/gitbutler#3512

Pre-commit hooks

• View hook failure logs gitbutlerapp/gitbutler#2551
• View hook failure logs gitbutlerapp/gitbutler#2551

Known Issues

• Case-insenstive filenames and renames are not supported gitbutlerapp/gitbutler#8793
• Unable to add any repository with fresh install of GitButler on MacOS - Unexpected token error gitbutlerapp/gitbutler#8948
• Bug: git_status shows non-existing untracked file starship/starship#6881 (comment)
• Trust-check doesn't always work on Windows #2128
• Simple commit traversal with hidden tips doesn't work at all with single-parent mode. #2159
• Simple commit traversal doesn't hide commits when using breadth-first traversal #2161

[darleybarreto]

Hey Sebastian,

Thanks for this! I was planning on doing this discovery later this week, because I'm off due to Carnival. I actually opened a discussion in the repo.

[Byron]

Thanks for kicking this off and for pointing to the original discussion! It helped me to understand what's going on as I was a little confused - now I see that a migration is happening to the -git crate and all makes sense.

I was pretty quick in creating this issue and I welcome you to double-check, or ask for clarification where you see fit or need. This can only make it better.

[krlvi]

Thank you @Byron for this! It was such a treat meeting you, and I am so excited about oxidising GitButler :)
I fully agree about the general principles that you outlined, it's very much how we like to approach problems too.

[Qix-]

add ODB alternates

While this is something Gitoxide should of course support, we're actually moving away from needing this long-term.

on Windows, there is no native ssh transport yet which might be worse than git2 right now.

Check out https://docs.rs/russh - they've been great maintainers and I used to use this crate for testing SSH stuff locally. Might work as a native transport. It's a maintained, spruced up version of Pijul's thrussh.

[Byron]

Thanks for the suggestions! I have just linked the gitoxide native #1246 issue, and believe that also lists russh as very strong candidate.

[Caleb-T-Owens]

Nice-to-haves that would be nice-to-have:

• EntryRef#kind() to alias EntryRef#mode().kind()
• Converting gix::object:tree::iter::EntryRef to a gix::object::tree::Entry is a three step nightmare. I've spent 15 minutes figuring this one out

                // Convert a gix::object::tree::iter::EntryRef to a gix::object::tree::Entry
                let detached = sub_entry.detach();
                let entry_detached: gix::objs::tree::Entry = detached.into();
                let entry = entry_detached.attach(sub_entry.repo);

[Byron]

Thanks, and even though I am usually hesitant to offer multiple paths to the same 'thing', this shortcut is unintuitive to not have. Please see #1975, which will be merged after collecting a couple more small changes.

[Caleb-T-Owens]

Thanks, and even though I am usually hesitant to offer multiple paths to the same 'thing', this shortcut is unintuitive to not have. Please see #1975, which will be merged after collecting a couple more small changes.

As an end user, I almost always want to work with higher level things and if needs be drop down to lower level abstractions.

To me, going SourceObject -> AbstractionOverProperty -> RawProperty would make sense as an accessing pattern for my use cases.

I guess the reason not to do that is if there is there might be an extra cost to producing AbstractionOverProperty or is potentially error prone, which is why gix goes SourceObject -> RawProperty -> AbstractionOverProperty, but it often leaves questions about how to actually get the things that I want to use

[Byron]

Thanks! With #1975 merged, both points mentioned in the comment above should be addressed. Point 2) added to_owned() , so one can do let entry = sub_entry.to_owned(), a missing .

And please, keep this feedback coming.

[Caleb-T-Owens]

Thank you for looking after these so quickly!