Use git diff for diffs (#150)

* NOTES

* Working on diff conversion

* Parsing diffs!

* Support num_lines for end skips

* Rename + Change test

* Test with patch-u5

* LocalFileDiff --> dataclass

* Codes --> dataclass

* Parsing raw diff output

* Use git diff --raw for directory diffs

* add diff endpoint

* Use git diff ops in webdiff!

* Context

* Update highlight.js

* post to /diff

* Add an option (-w)

* Factor out DiffOptionsControl

* wire up context + diff algorithm

* Add git diff -b as well; unclear if this is different?

* styled diff options panel

* thread through some options

* wire up styling

* Wire up configurable highlight.js themes

* fix "show -3 lines" bug

* maxLinesForSyntax

* Support binary diffs again

* update codediff.js, bugfix

* Fix bug with pure moves

* add unidiff dependency

* delete irrelevant test

* blacken, version bump

* Fix issue with git webdiff by adding --no-symlinks

* blacken

* check for symlinks and inline

* we may be in business!

* swap in symlink-y directory

* test case for "Show -1 more lines"

* no chardiff on last line

* no-trailing-newline test

* fix bug w/ trailing newlines

* pin Werkzeug

* delete TODO

* add long line no space test

* pare back test case

* overflow-wrap: anywhere

* webdiff.colors

* add docs on configuration

* fix typo

* add some implementation notes

Author: danvk

.gitignore

@@ -64,3 +64,4 @@ webdiff/static/js/file_diff.js
 
 wheelhouse
 .vscode
+NOTES.md

README

@@ -4,6 +4,7 @@
 Two-column web-based git difftool.
 
 Features include:
+
 * Side-by-side (two column) diff view
 * Runs in the browser of your choice on any platform.
 * Syntax highlighting via highlight.js
@@ -60,6 +61,33 @@ Make sure you chmod this file to only be readable by yourself. You can generate
 a personal access token for webdiff via github.com → profile → Settings →
 Personal access tokens. Make sure to grant all the "repo" privileges.
 
+## Configuration
+
+webdiff can be configured via [`git config`][git config]. To change the syntax highlighting theme, for example:
+
+    git config webdiff.theme rainbow
+
+(You can find a list of supported themes in the [themes] directory.)
+
+As with any git configuration setting, these can be set globally or per-repo.
+
+Options are:
+
+| Setting        | Default       | Notes  |
+| -------------- | ------------- | ------ |
+| webdiff.theme  | googlecode    | Syntax highlighting theme (see [themes] directory). |
+| webdiff.port   | -1            | Port on which to serve webdiff. Default is random open port. This can be overridden with `--port`. |
+| webdiff.maxDiffWidth | 100 | Maximum length of lines in the diff display. After this width, lines will wrap. |
+| webdiff.unified | 8 | Lines of context to display by default (`git diff -U`) |
+| webdiff.extraDirDiffArgs | "" | Any extra arguments to pass to `git diff` when diffing directories. |
+| webdiff.extraFileDiffArgs | "" | Any extra arguments to pass to `git diff` when diffing files. |
+| webdiff.openBrowser | true | Whether to automatically open the browser UI when you run webdiff. |
+| webdiff.maxLinesForSyntax | 10000 | Maximum lines in file to do syntax highlighting. |
+| webdiff.colors.delete | #fee | CSS background color for delete (left) lines |
+| webdiff.colors.insert | #efe | CSS background color for insert (right) lines |
+| webdiff.colors.charDelete | #fcc | CSS background color for deleted characters in a delete (left) line |
+| webdiff.colors.charInsert | #cfc | CSS background color for inserted characters in an insert (right) line |
+
 ## Development
 
     python3 -m venv venv
@@ -76,38 +104,37 @@ Then from the root directory:
 
 or to launch in debug mode:
 
-    ./test.sh $(pwd)/../testdata/webdiffdiff/{left,right}
+    ./test.sh $(pwd)/testdata/manyfiles/{left,right}
 
 (or any other directory in testdata)
 
 To run the Python tests:
 
     pytest
 
-To run the JavaScript tests:
-
-    python -m SimpleHTTPServer
-    open tests/runner.html
-
 To format the code, run:
 
     ./scripts/black.sh
     cd ts
     yarn prettier
 
+To debug `git webdiff`, run:
+
+    WEBDIFF_CONFIG=$(pwd)/testing.cfg ./webdiff/gitwebdiff.py
+
 To iterate on the PyPI package, run:
 
     # from outside the webdiff virtualenv:
-    pip uninstall webdiff
+    pip3 uninstall webdiff
 
     # from inside the webdiff virtualenv, adjust for current version
     python setup.py sdist
     mkdir /tmp/webdiff-test
-    cp dist/webdiff-X.Y.Z.tar.gz /tmp/webdiff-test
+    cp dist/webdiff-?.?.?.tar.gz /tmp/webdiff-test
 
     deactivate
     cd /tmp/webdiff-test
-    pip3 install webdiff-X.Y.Z.tar.gz
+    pip3 install webdiff-?.?.?.tar.gz
 
 To publish to pypitest:
 
@@ -121,6 +148,30 @@ And to the real pypi:
 
 See [pypirc][] docs for details on setting up `~/.pypirc`.
 
+## Implementation notes
+
+webdiff doesn't calculate any diffs itself. Instead, it relies on `git diff`. This is possible because `git diff` has a `--no-index` mode that allows it to operate outside of a git repository. Of course, this means that you need to have `git` installed to use webdiff!
+
+When you run `webdiff dir1 dir2`, webdiff runs:
+
+    git diff --raw --no-index dir1 dir2
+
+To ask `git` which files are adds, removes, renames and changes. Then, when it's serving the web UI for a particular diff, it runs:
+
+    git diff --no-index (diff args) file1 file2
+
+This produces a patch, which is what the web UI renders. (It also needs both full files for syntax highlighting.)
+
+When you run `git webdiff (args)`, it runs:
+
+    git difftool -d -x webdiff (args)
+
+This tells `git` to set up two directories and invoke `webdiff leftdir rightdir`.
+
+There's one complication involving symlinks. `git difftool -d` may fill one of the sides (typically the right) with symlinks. This is faster than copying files, but unfortunately `git diff --no-index` does not resolve these symlinks. To make this work, if a directory contains symlinks, webdiff makes a copy of it before diffing. For file diffs, it resolves the symlink before passing it to `git diff --no-index`. The upshot is that you can run `git webdiff`, edit a file, reload the browser window and see the changes.
+
 [pypirc]: https://packaging.python.org/specifications/pypirc/
 [Homebrew]: https://brew.sh/
 [ImageMagick]: https://imagemagick.org/index.php
+[git config]: https://git-scm.com/docs/git-config
+[themes]: http://example.com

README.md

@@ -5,3 +5,5 @@ pillow
 requests
 binaryornot
 black
+unidiff==0.7.4
+Werkzeug==2.2.2

TODO

@@ -6,7 +6,7 @@
 
 
 setup(name='webdiff',
-      version='0.16.0',
+      version='1.0.0',
       description='Two-column web-based git difftool',
       long_description=long_description,
       long_description_content_type='text/markdown',
@@ -25,7 +25,9 @@
           'flask==2.2.2',
           'pillow',
           'requests',
-          'PyGithub==1.55'
+          'PyGithub==1.55',
+          'unidiff==0.7.4',
+          'Werkzeug==2.2.2',
       ],
       include_package_data=True,
       package_data = {

requirements.txt

@@ -0,0 +1,7 @@
+#!/bin/bash
+export TESTING=true
+export DEBUG=true
+export WEBDIFF_CONFIG=$(pwd)/testing.cfg
+export WEBDIFF_PORT=$(($RANDOM + 10000))
+export PYTHONPATH=.
+./webdiff/gitwebdiff.py $*

setup.py

@@ -0,0 +1,5 @@
+Now it's completely transparent that `requestStatus` will end in "success." It's easy to accidentally produce half-synchronous code with callbacks or raw Promises, but difficult with `async`.footnote:[There's still a more subtle bug in this version: if you call fetchWithCache twice in a row with the same URL, it will issue two requests. How would you fix this?]
+
+// This is the fix, it's three lines longer but I think it adds complexity that's irrelevant to the point that I'm trying to make here. https://www.typescriptlang.org/play?#code/MYewdgzgLgBA+sAhsAFgUwFwwN4G0CuATgDZbSECWYA5gLpYAKhIAthRGgDwBKaEADuA4A+AL4wAvDlEBuAFCIIATzDAYAM3yqoFcBrRRUAdQpQUAYWToAFEVIxyVagEpGzNhx59BkNMJxyMDAU6jC2JMFg8EioaM4BQUGEBkRRCFZoBCS08kGigTCgkLDqBqgMkvqGKOHEzrnRGVnEtJWl1QwNyVCpVeXy+YoqapraulHtqAAqaAAeUCZmlrG1ZFCUNK4wTKzsXI40-tgFRdAwyQJCaJWIAO6Ipn0oixYZtfUF3b0XPhwAdFA5lBrB98nJiAZzmgAI74PhQADKUEQPQgWAA5MQQIgACZOdEwAA+MHREHwwGAfAgBOJ6LQhGYhHR8iGqg0WmAOj01AMAFUOIRbAKAJI4tYbFwJKGw+FIlH4CCVTHYvE0ZknISwfjMdQUCE3e6PSYoGbzF7LGwAAwA9Ar6daACTYO2EUWiS0fJIwuHQOWopVkilU9X5IA
+
+Note that if you return a Promise from an `async` function, it will not get wrapped in another Promise: the return type will be `Promise<T>` rather than `Promise<Promise<T>>`. Again, TypeScript will help you build an intuition for this:

test-gitwebdiff.sh

@@ -0,0 +1,5 @@
+Now it's completely transparent that `requestStatus` will end in "success." It's easy to accidentally produce half-synchronous code with callbacks or raw Promises, but difficult with `async`.footnote:[There's still a more subtle bug in this version: if you call ++fetchWithCache++ twice in a row with the same URL, it will issue two requests. How would you fix this?]
+
+// This is the fix, it's three lines longer but I think it adds complexity that's irrelevant to the point that I'm trying to make here. https://www.typescriptlang.org/play?#code/MYewdgzgLgBA+sAhsAFgUwFwwN4G0CuATgDZbSECWYA5gLpYAKhIAthRGgDwBKaEADuA4A+AL4wAvDlEBuAFCIIATzDAYAM3yqoFcBrRRUAdQpQUAYWToAFEVIxyVagEpGzNhx59BkNMJxyMDAU6jC2JMFg8EioaM4BQUGEBkRRCFZoBCS08kGigTCgkLDqBqgMkvqGKOHEzrnRGVnEtJWl1QwNyVCpVeXy+YoqapraulHtqAAqaAAeUCZmlrG1ZFCUNK4wTKzsXI40-tgFRdAwyQJCaJWIAO6Ipn0oixYZtfUF3b0XPhwAdFA5lBrB98nJiAZzmgAI74PhQADKUEQPQgWAA5MQQIgACZOdEwAA+MHREHwwGAfAgBOJ6LQhGYhHR8iGqg0WmAOj01AMAFUOIRbAKAJI4tYbFwJKGw+FIlH4CCVTHYvE0ZknISwfjMdQUCE3e6PSYoGbzF7LGwAAwA9Ar6daACTYO2EUWiS0fJIwuHQOWopVkilU9X5IA
+
+Note that if you return a Promise from an `async` function, it will not get wrapped in another Promise: the return type will be `Promise<T>` rather than ++Promise&#x200b;&lt;Promise&lt;T&gt;&gt;++. Again, TypeScript will help you build an intuition for this:

testdata/longline-nospace/left/text.asciidoc

@@ -0,0 +1,4 @@
+This is a file
+It has four lines
+and a
+trailing newline