|
| 1 | +Use of index and Racy git problem |
| 2 | +================================= |
| 3 | + |
| 4 | +Background |
| 5 | +---------- |
| 6 | + |
| 7 | +The index is one of the most important data structure in git. |
| 8 | +It represents a virtual working tree state by recording list of |
| 9 | +paths and their object names and serves as a staging area to |
| 10 | +write out the next tree object to be committed. The state is |
| 11 | +"virtual" in the sense that it does not necessarily have to, and |
| 12 | +often does not, match the files in the working tree. |
| 13 | + |
| 14 | +There are cases git needs to examine the differences between the |
| 15 | +virtual working tree state in the index and the files in the |
| 16 | +working tree. The most obvious case is when the user asks `git |
| 17 | +diff` (or its low level implementation, `git diff-files`) or |
| 18 | +`git-ls-files --modified`. In addition, git internally checks |
| 19 | +if the files in the working tree is different from what are |
| 20 | +recorded in the index to avoid stomping on local changes in them |
| 21 | +during patch application, switching branches, and merging. |
| 22 | + |
| 23 | +In order to speed up this comparison between the files in the |
| 24 | +working tree and the index entries, the index entries record the |
| 25 | +information obtained from the filesystem via `lstat(2)` system |
| 26 | +call when they were last updated. When checking if they differ, |
| 27 | +git first runs `lstat(2)` on the files and compare the result |
| 28 | +with this information (this is what was originally done by the |
| 29 | +`ce_match_stat()` function, which the current code does in |
| 30 | +`ce_match_stat_basic()` function). If some of these "cached |
| 31 | +stat information" fields do not match, git can tell that the |
| 32 | +files are modified without even looking at their contents. |
| 33 | + |
| 34 | +Note: not all members in `struct stat` obtained via `lstat(2)` |
| 35 | +are used for this comparison. For example, `st_atime` obviously |
| 36 | +is not useful. Currently, git compares the file type (regular |
| 37 | +files vs symbolic links) and executable bits (only for regular |
| 38 | +files) from `st_mode` member, `st_mtime` and `st_ctime` |
| 39 | +timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members. |
| 40 | +With a `USE_STDEV` compile-time option, `st_dev` is also |
| 41 | +compared, but this is not enabled by default because this member |
| 42 | +is not stable on network filesystems. With `USE_NSEC` |
| 43 | +compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec` |
| 44 | +members are also compared, but this is not enabled by default |
| 45 | +because the value of this member becomes meaningless once the |
| 46 | +inode is evicted from the inode cache on filesystems that do not |
| 47 | +store it on disk. |
| 48 | + |
| 49 | + |
| 50 | +Racy git |
| 51 | +-------- |
| 52 | + |
| 53 | +There is one slight problem with the optimization based on the |
| 54 | +cached stat information. Consider this sequence: |
| 55 | + |
| 56 | + $ git update-index 'foo' |
| 57 | + : modify 'foo' in-place without changing its size |
| 58 | + |
| 59 | +The first `update-index` computes the object name of the |
| 60 | +contents of file `foo` and updates the index entry for `foo` |
| 61 | +along with the `struct stat` information. If the modification |
| 62 | +that follows it happens very fast so that the file's `st_mtime` |
| 63 | +timestamp does not change, after this sequence, the cached stat |
| 64 | +information the index entry records still exactly match what you |
| 65 | +can obtain from the filesystem, but the file `foo` is modified. |
| 66 | +This way, git can incorrectly think files in the working tree |
| 67 | +are unmodified even though they actually are. This is called |
| 68 | +the "racy git" problem (discovered by Pasky), and the entries |
| 69 | +that appear clean when they may not be because of this problem |
| 70 | +are called "racily clean". |
| 71 | + |
| 72 | +To avoid this problem, git does two things: |
| 73 | + |
| 74 | +. When the cached stat information says the file has not been |
| 75 | + modified, and the `st_mtime` is the same as (or newer than) |
| 76 | + the timestamp of the index file itself (which is the time `git |
| 77 | + update-index foo` finished running in the above example), it |
| 78 | + also compares the contents with the object registered in the |
| 79 | + index entry to make sure they match. |
| 80 | + |
| 81 | +. When the index file is updated that contains racily clean |
| 82 | + entries, cached `st_size` information is truncated to zero |
| 83 | + before writing a new version of the index file. |
| 84 | + |
| 85 | +Because the index file itself is written after collecting all |
| 86 | +the stat information from updated paths, `st_mtime` timestamp of |
| 87 | +it is usually the same as or newer than any of the paths the |
| 88 | +index contains. And no matter how quick the modification that |
| 89 | +follows `git update-index foo` finishes, the resulting |
| 90 | +`st_mtime` timestamp on `foo` cannot get the timestamp earlier |
| 91 | +than the index file. Therefore, index entries that can be |
| 92 | +racily clean are limited to the ones that have the same |
| 93 | +timestamp as the index file itself. |
| 94 | + |
| 95 | +The callers that want to check if an index entry matches the |
| 96 | +corresponding file in the working tree continue to call |
| 97 | +`ce_match_stat()`, but with this change, `ce_match_stat()` uses |
| 98 | +`ce_modified_check_fs()` to see if racily clean ones are |
| 99 | +actually clean after comparing the cached stat information using |
| 100 | +`ce_match_stat_basic()`. |
| 101 | + |
| 102 | +The problem the latter solves is this sequence: |
| 103 | + |
| 104 | + $ git update-index 'foo' |
| 105 | + : modify 'foo' in-place without changing its size |
| 106 | + : wait for enough time |
| 107 | + $ git update-index 'bar' |
| 108 | + |
| 109 | +Without the latter, the timestamp of the index file gets a newer |
| 110 | +value, and falsely clean entry `foo` would not be caught by the |
| 111 | +timestamp comparison check done with the former logic anymore. |
| 112 | +The latter makes sure that the cached stat information for `foo` |
| 113 | +would never match with the file in the working tree, so later |
| 114 | +checks by `ce_match_stat_basic()` would report the index entry |
| 115 | +does not match the file and git does not have to fall back on more |
| 116 | +expensive `ce_modified_check_fs()`. |
| 117 | + |
| 118 | + |
| 119 | +Runtime penalty |
| 120 | +--------------- |
| 121 | + |
| 122 | +The runtime penalty of falling back to `ce_modified_check_fs()` |
| 123 | +from `ce_match_stat()` can be very expensive when there are many |
| 124 | +racily clean entries. An obvious way to artificially create |
| 125 | +this situation is to give the same timestamp to all the files in |
| 126 | +the working tree in a large project, run `git update-index` on |
| 127 | +them, and give the same timestamp to the index file: |
| 128 | + |
| 129 | + $ date >.datestamp |
| 130 | + $ git ls-files | xargs touch -r .datestamp |
| 131 | + $ git ls-files | git update-index --stdin |
| 132 | + $ touch -r .datestamp .git/index |
| 133 | + |
| 134 | +This will make all index entries racily clean. The linux-2.6 |
| 135 | +project, for example, there are over 20,000 files in the working |
| 136 | +tree. On my Athron 64X2 3800+, after the above: |
| 137 | + |
| 138 | + $ /usr/bin/time git diff-files |
| 139 | + 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k |
| 140 | + 0inputs+0outputs (0major+67111minor)pagefaults 0swaps |
| 141 | + $ git update-index MAINTAINERS |
| 142 | + $ /usr/bin/time git diff-files |
| 143 | + 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k |
| 144 | + 0inputs+0outputs (0major+935minor)pagefaults 0swaps |
| 145 | + |
| 146 | +Running `git update-index` in the middle checked the racily |
| 147 | +clean entries, and left the cached `st_mtime` for all the paths |
| 148 | +intact because they were actually clean (so this step took about |
| 149 | +the same amount of time as the first `git diff-files`). After |
| 150 | +that, they are not racily clean anymore but are truly clean, so |
| 151 | +the second invocation of `git diff-files` fully took advantage |
| 152 | +of the cached stat information. |
| 153 | + |
| 154 | + |
| 155 | +Avoiding runtime penalty |
| 156 | +------------------------ |
| 157 | + |
| 158 | +In order to avoid the above runtime penalty, the recent "master" |
| 159 | +branch (post 1.4.2) has a code that makes sure the index file |
| 160 | +gets timestamp newer than the youngest files in the index when |
| 161 | +there are many young files with the same timestamp as the |
| 162 | +resulting index file would otherwise would have by waiting |
| 163 | +before finishing writing the index file out. |
| 164 | + |
| 165 | +I suspect that in practice the situation where many paths in the |
| 166 | +index are all racily clean is quite rare. The only code paths |
| 167 | +that can record recent timestamp for large number of paths I |
| 168 | +know of are: |
| 169 | + |
| 170 | +. Initial `git add .` of a large project. |
| 171 | + |
| 172 | +. `git checkout` of a large project from an empty index into an |
| 173 | + unpopulated working tree. |
| 174 | + |
| 175 | +Note: switching branches with `git checkout` keeps the cached |
| 176 | +stat information of existing working tree files that are the |
| 177 | +same between the current branch and the new branch, which are |
| 178 | +all older than the resulting index file, and they will not |
| 179 | +become racily clean. Only the files that are actually checked |
| 180 | +out can become racily clean. |
| 181 | + |
| 182 | +In a large project where raciness avoidance cost really matters, |
| 183 | +however, the initial computation of all object names in the |
| 184 | +index takes more than one second, and the index file is written |
| 185 | +out after all that happens. Therefore the timestamp of the |
| 186 | +index file will be more than one seconds later than the the |
| 187 | +youngest file in the working tree. This means that in these |
| 188 | +cases there actually will not be any racily clean entry in |
| 189 | +the resulting index. |
| 190 | + |
| 191 | +So in summary I think we should not worry about avoiding the |
| 192 | +runtime penalty and get rid of the "wait before finishing |
| 193 | +writing" code out. |
0 commit comments