olsp can keep incorrect document src in its state because of encoding issues

[ulugbekna]

olsp sometimes formats code into illegal state, e.g., given file

let foo ~i ~j = i + j
let bar = 10
let y = foo ~i:5

let lst = []

let map = List.m

let map = List.m []

let map = List.m

olsp formats it to

let foo ~i ~j = i + j
let bar = 10
let y = foo ~i:5
let lst = []
let map = List.mlet map = List.m []
let map = List.m

trace (notice how ofmt says it's disabled but still formats code)

[Trace - 10:35:22 PM] Sending request 'textDocument/formatting - (374)'.
Params: {
    "textDocument": {
        "uri": "file:///Users/ulugbek/code/merlin/tests/test-dirs/completion/application_context.t/application_context.ml"
    },
    "options": {
        "tabSize": 2,
        "insertSpaces": true
    }
}


File tests/test-dirs/completion/application_context.t/application_context.ml
Warning: Ocamlformat disabled because [--enable-outside-detected-project] is not set and no [.ocamlformat] was found within the project (root: ../merlin)
[Trace - 10:35:22 PM] Received response 'textDocument/formatting - (374)' in 3ms.
Result: [
    {
        "newText": "",
        "range": {
            "end": {
                "character": 0,
                "line": 4
            },
            "start": {
                "character": 0,
                "line": 3
            }
        }
    },
    {
        "newText": "",
        "range": {
            "end": {
                "character": 0,
                "line": 10
            },
            "start": {
                "character": 0,
                "line": 5
            }
        }
    },
    {
        "newText": "let map = List.m []\nlet map = List.m\n",
        "range": {
            "end": {
                "character": 0,
                "line": 11
            },
            "start": {
                "character": 0,
                "line": 11
            }
        }
    }
]

---

ofmt also sometimes gets stuck and can't get format; restarting olsp enables it to format again.

---

this must be some ofmt-rpc issue

[rgrinberg]

Shall we disable ocamlformat-rpc if it's not stable enough?

[ulugbekna]

I think we should disable it for formatting files. Seems fine enough for code snippets on hover, etc

[ulugbekna]

The rpc also doesn't respect a missing .ocamlformat in the project and formats files with default config, which is unacceptable for us

[ulugbekna]

I have a suspicion that legal code not being formatted is actually caused by our representation of document source code because I think I see "phantom" syntax errors (ie source is legit in the editor but there's a syntax error shown for strange position within the source). It could be the encoding or batching document updates PR at fault. #940 would make it easier to debug.

[rgrinberg]

I have a feeling that this bug was here all along. In fact, since the encoding PR, I haven't been able to reproduce it. Perhaps because neovim allows for utf8. It might be the bug is in the utf16 decoding?

[ulugbekna]

have a feeling that this bug was here all along.

interesting. I never had this problem before, at least not past year.

Indeed, vscode uses utf16 position encoding, so that may be the problem