Merge branch 'scalar'

This merges the upstreamable part of the Scalar patches.

Minor merge conflicts (caused by the gvfs-helper) were resolved
trivially.

Signed-off-by: Johannes Schindelin <[email protected]>

Author: dscho

Documentation/config/core.adoc

@@ -828,3 +828,12 @@ core.WSLCompat::
 	The default value is false. When set to true, Git will set the mode
 	bits of the file in the way of wsl, so that the executable flag of
 	files can be set or read correctly.
+
+core.configWriteLockTimeoutMS::
+	When processes try to write to the config concurrently, it is likely
+	that one process "wins" and the other process(es) fail to lock the
+	config file. By configuring a timeout larger than zero, Git can be
+	told to try to lock the config again a couple times within the
+	specified timeout. If the timeout is configure to zero (which is the
+	default), Git will fail immediately when the config is already
+	locked.

config.c

@@ -3271,6 +3271,7 @@ int repo_config_set_multivar_in_file_gently(struct repository *r,
 					    const char *comment,
 					    unsigned flags)
 {
+	static unsigned long timeout_ms = ULONG_MAX;
 	int fd = -1, in_fd = -1;
 	int ret;
 	struct lock_file lock = LOCK_INIT;
@@ -3291,11 +3292,16 @@ int repo_config_set_multivar_in_file_gently(struct repository *r,
 	if (!config_filename)
 		config_filename = filename_buf = repo_git_path(r, "config");
 
+	if ((long)timeout_ms < 0 &&
+	    git_config_get_ulong("core.configWriteLockTimeoutMS", &timeout_ms))
+		timeout_ms = 0;
+
 	/*
 	 * The lock serves a purpose in addition to locking: the new
 	 * contents of .git/config will be written into it.
 	 */
-	fd = hold_lock_file_for_update(&lock, config_filename, 0);
+	fd = hold_lock_file_for_update_timeout(&lock, config_filename, 0,
+					       timeout_ms);
 	if (fd < 0) {
 		error_errno(_("could not lock config file %s"), config_filename);
 		ret = CONFIG_NO_LOCK;

contrib/scalar/docs/faq.md

@@ -0,0 +1,51 @@
+Frequently Asked Questions
+==========================
+
+Using Scalar
+------------
+
+### I don't want a sparse clone, I want every file after I clone!
+
+Run `scalar clone --full-clone <url>` to initialize your repo to include
+every file. You can switch to a sparse-checkout later by running
+`git sparse-checkout init --cone`.
+
+### I already cloned without `--full-clone`. How do I get everything?
+
+Run `git sparse-checkout disable`.
+
+Scalar Design Decisions
+-----------------------
+
+There may be many design decisions within Scalar that are confusing at first
+glance. Some of them may cause friction when you use Scalar with your existing
+repos and existing habits.
+
+> Scalar has the most benefit when users design repositories
+> with efficient patterns.
+
+For example: Scalar uses the sparse-checkout feature to limit the size of the
+working directory within a large monorepo. It is designed to work efficiently
+with monorepos that are highly componentized, allowing most developers to
+need many fewer files in their daily work.
+
+### Why does `scalar clone` create a `<repo>/src` folder?
+
+Scalar uses a file system watcher to keep track of changes under this `src` folder.
+Any activity in this folder is assumed to be important to Git operations. By
+creating the `src` folder, we are making it easy for your build system to
+create output folders outside the `src` directory. We commonly see systems
+create folders for build outputs and package downloads. Scalar itself creates
+these folders during its builds.
+
+Your build system may create build artifacts such as `.obj` or `.lib` files
+next to your source code. These are commonly "hidden" from Git using
+`.gitignore` files. Having such artifacts in your source tree creates
+additional work for Git because it needs to look at these files and match them
+against the `.gitignore` patterns.
+
+By following the `src` pattern Scalar tries to establish and placing your build
+intermediates and outputs parallel with the `src` folder and not inside it,
+you can help optimize Git command performance for developers in the repository
+by limiting the number of files Git needs to consider for many common
+operations.

contrib/scalar/docs/getting-started.md

@@ -0,0 +1,93 @@
+Getting Started
+===============
+
+Registering existing Git repos
+------------------------------
+
+To add a repository to the list of registered repos, run `scalar register [<path>]`.
+If `<path>` is not provided, then the "current repository" is discovered from
+the working directory by scanning the parent paths for a path containing a `.git`
+folder, possibly inside a `src` folder.
+
+To see which repositories are currently tracked by the service, run
+`scalar list`.
+
+Run `scalar unregister [<path>]` to remove the repo from this list.
+
+Creating a new Scalar clone
+---------------------------------------------------
+
+The `clone` verb creates a local enlistment of a remote repository using the
+partial clone feature available e.g. on GitHub.
+
+
+```
+scalar clone [options] <url> [<dir>]
+```
+
+Create a local copy of the repository at `<url>`. If specified, create the `<dir>`
+directory and place the repository there. Otherwise, the last section of the `<url>`
+will be used for `<dir>`.
+
+At the end, the repo is located at `<dir>/src`. By default, the sparse-checkout
+feature is enabled and the only files present are those in the root of your
+Git repository. Use `git sparse-checkout set` to expand the set of directories
+you want to see, or `git sparse-checkout disable` to expand to all files. You
+can explore the subdirectories outside your sparse-checkout specification using
+`git ls-tree HEAD`.
+
+### Sparse Repo Mode
+
+By default, Scalar reduces your working directory to only the files at the
+root of the repository. You need to add the folders you care about to build up
+to your working set.
+
+* `scalar clone <url>`
+  * Please choose the **Clone with HTTPS** option in the `Clone Repository` dialog in Azure Repos, not **Clone with SSH**.
+* `cd <root>\src`
+* At this point, your `src` directory only contains files that appear in your root
+  tree. No folders are populated.
+* Set the directory list for your sparse-checkout using:
+	1. `git sparse-checkout set <dir1> <dir2> ...`
+	2. `git sparse-checkout set --stdin < dir-list.txt`
+* Run git commands as you normally would.
+* To fully populate your working directory, run `git sparse-checkout disable`.
+
+If instead you want to start with all files on-disk, you can clone with the
+`--full-clone` option. To enable sparse-checkout after the fact, run
+`git sparse-checkout init --cone`. This will initialize your sparse-checkout
+patterns to only match the files at root.
+
+If you are unfamiliar with what directories are available in the repository,
+then you can run `git ls-tree -d --name-only HEAD` to discover the directories
+at root, or `git ls-tree -d --name-only HEAD <path>` to discover the directories
+in `<path>`.
+
+### Options
+
+These options allow a user to customize their initial enlistment.
+
+* `--full-clone`: If specified, do not initialize the sparse-checkout feature.
+  All files will be present in your `src` directory. This uses a Git partial
+  clone: blobs are downloaded on demand.
+
+* `--branch=<ref>`: Specify the branch to checkout after clone.
+
+### Advanced Options
+
+The options below are not intended for use by a typical user. These are
+usually used by build machines to create a temporary enlistment that
+operates on a single commit.
+
+* `--single-branch`: Use this option to only download metadata for the branch
+  that will be checked out. This is helpful for build machines that target
+  a remote with many branches. Any `git fetch` commands after the clone will
+  still ask for all branches.
+
+Removing a Scalar Clone
+-----------------------
+
+Since the `scalar clone` command sets up a file-system watcher (when available),
+that watcher could prevent deleting the enlistment. Run `scalar delete <path>`
+from outside of your enlistment to unregister the enlistment from the filesystem
+watcher and delete the enlistment at `<path>`.

contrib/scalar/docs/index.md

@@ -0,0 +1,50 @@
+Scalar: Enabling Git at Scale
+=============================
+
+Scalar is a tool that helps Git scale to some of the largest Git repositories.
+It achieves this by enabling some advanced Git features, such as:
+
+* *Partial clone:* reduces time to get a working repository by not
+  downloading all Git objects right away.
+
+* *Background prefetch:* downloads Git object data from all remotes every
+  hour, reducing the amount of time for foreground `git fetch` calls.
+
+* *Sparse-checkout:* limits the size of your working directory.
+
+* *File system monitor:* tracks the recently modified files and eliminates
+  the need for Git to scan the entire worktree.
+
+* *Commit-graph:* accelerates commit walks and reachability calculations,
+   speeding up commands like `git log`.
+
+* *Multi-pack-index:* enables fast object lookups across many pack-files.
+
+* *Incremental repack:* Repacks the packed Git data into fewer pack-file
+  without disrupting concurrent commands by using the multi-pack-index.
+
+By running `scalar register` in any Git repo, Scalar will automatically enable
+these features for that repo (except partial clone) and start running suggested
+maintenance in the background using
+[the `git maintenance` feature](https://git-scm.com/docs/git-maintenance).
+
+Repos cloned with the `scalar clone` command use partial clone to significantly
+reduce the amount of data required to get started using a repository. By
+delaying all blob downloads until they are required, Scalar allows you to work
+with very large repositories quickly.
+
+Documentation
+-------------
+
+* [Getting Started](getting-started.md): Get started with Scalar.
+  Includes `scalar register`, `scalar unregister`, `scalar clone`, and
+  `scalar delete`.
+
+* [Troubleshooting](troubleshooting.md):
+  Collect diagnostic information or update custom settings. Includes
+  `scalar diagnose`.
+
+* [The Philosophy of Scalar](philosophy.md): Why does Scalar work the way
+  it does, and how do we make decisions about its future?
+
+* [Frequently Asked Questions](faq.md)

contrib/scalar/docs/philosophy.md

@@ -0,0 +1,66 @@
+The Philosophy of Scalar
+========================
+
+The team building Scalar has **opinions** about Git performance. Scalar
+takes out the guesswork by automatically configuring your Git repositories
+to take advantage of the latest and greatest features. It is difficult to
+say that these are the absolute best settings for every repository, but
+these settings do work for some of the largest repositories in the world.
+
+Scalar intends to do very little more than the standard Git client. We
+actively implement new features into Git instead of Scalar, then update
+Scalar only to configure those new settings. In particular, we ported
+features like background maintenance to Git to make Scalar simpler and
+make Git more powerful.
+
+Services such as GitHub support partial clone , a standard adopted by the Git
+project to download only part of the Git objects when cloning, and fetching
+further objects on demand. If your hosting service supports partial clone, then
+we absolutely recommend it as a way to greatly speed up your clone and fetch
+times and to reduce how much disk space your Git repository requires. Scalar
+will help with this!
+
+Most of the value of Scalar can be found in the core Git client. However, most
+of the advanced features that really optimize Git's performance are off by
+default for compatibility reasons. To really take advantage of Git's latest and
+greatest features, you either need to study the [`git config`
+documentation](https://git-scm.com/docs/git-config) and regularly read [the Git
+release notes](https://github.com/git/git/tree/master/Documentation/RelNotes).
+Even if you do all that work and customize your Git settings on your machines,
+you likely will want to share those settings with other team members. Or, you
+can just use Scalar!
+
+Using `scalar register` on an existing Git repository will give you these
+benefits:
+
+* Additional compression of your `.git/index` file.
+* Hourly background `git fetch` operations, keeping you in-sync with your
+  remotes.
+* Advanced data structures, such as the `commit-graph` and `multi-pack-index`
+  are updated automatically in the background.
+* If using macOS or Windows, then Scalar configures Git's builtin File System
+  Monitor, providing faster commands such as `git status` or `git add`.
+
+Additionally, if you use `scalar clone` to create a new repository, then
+you will automatically get these benefits:
+
+* Use Git's partial clone feature to only download the files you need for
+  your current checkout.
+* Use Git's [sparse-checkout feature][sparse-checkout] to minimize the
+  number of files required in your working directory.
+  [Read more about sparse-checkout here.][sparse-checkout-blog]
+* Create the Git repository inside `<repo-name>/src` to make it easy to
+  place build artifacts outside of the Git repository, such as in
+  `<repo-name>/bin` or `<repo-name>/packages`.
+
+We also admit that these **opinions** can always be improved! If you have
+an idea of how to improve our setup, consider
+[creating an issue](https://github.com/microsoft/scalar/issues/new) or
+contributing a pull request! Some [existing](https://github.com/microsoft/scalar/issues/382)
+[issues](https://github.com/microsoft/scalar/issues/388) have already
+improved our configuration settings and roadmap!
+
+[gvfs-protocol]: https://github.com/microsoft/VFSForGit/blob/HEAD/Protocol.md
+[microsoft-git]: https://github.com/microsoft/git
+[sparse-checkout]: https://git-scm.com/docs/git-sparse-checkout
+[sparse-checkout-blog]: https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/

contrib/scalar/docs/troubleshooting.md

@@ -0,0 +1,20 @@
+Troubleshooting
+===============
+
+Diagnosing Issues
+-----------------
+
+The `scalar diagnose` command collects logs and config details for the current
+repository. The resulting zip file helps root-cause issues.
+
+When run inside your repository, creates a zip file containing several important
+files for that repository. This includes:
+
+* Configuration files from your `.git` folder, such as the `config` file,
+  `index`, `hooks`, and `refs`.
+
+* A summary of your Git object database, including the number of loose objects
+  and the names and sizes of pack-files.
+
+As the `diagnose` command completes, it provides the path of the resulting
+zip file. This zip can be attached to bug reports to make the analysis easier.