Complex parser and post-processing

[vheon]

This might be a little convoluted but I'll try to explain the whole picture so that if there is a better way to do it, the better.

I work in C++ and for example in a source code like:

int square(int num) {
    return num * num;
}

int main()
{
    return squaree(3);
}

gcc would spit out this output:

main.cpp: In function 'int main()':
main.cpp:8:12: error: 'squaree' was not declared in this scope; did you mean 'square'?
    8 |     return squaree(3);
      |            ^~~~~~~
      |            square

What I wanted to do was to parse the whole error and I ended up with:

local problem_matcher = require("overseer.template.vscode.problem_matcher")
local gcc_parser = problem_matcher.get_parser_from_problem_matcher(problem_matcher.resolve_problem_matcher("$gcc"))
gcc_parser[2].append = false

-- components section of the template
components = {
    "defaults",
    {
        "on_output_parse",
        parser = {
            diagnostics = {
                "sequence",
                gcc_parser,
                { "extract_multiline", "%s+%d*%s*|(%s+.*)", "text" }
            }
        }
    },
},

Now with this, trouble.nvim and a custom configuration for Neovim's diagnostics to show only the first line I have a comfortable setup where diagnostics are shown, I can navigate them with trouble and see the whole message form gcc within trouble.

The first question here is about this:

local problem_matcher = require("overseer.template.vscode.problem_matcher")
local gcc_parser = problem_matcher.get_parser_from_problem_matcher(problem_matcher.resolve_problem_matcher("$gcc"))
gcc_parser[2].append = false

is this the correct way to do it? is there a better way?

The second question is around post processing of the lines in the parser in order to add more information to the diagnostics: what I mean is that I wanted to parse the amount of ~ in the diagnostics so that I could set the end_col of the diagnostic and basically reproduce what gcc spits out in the editor.

I couldn't find a way to do it reading the documentation (or maybe I missed the needed piece) so I thought of writing a parser that would do this custom thing mimicking the implementation of https://github.com/stevearc/overseer.nvim/blob/master/lua/overseer/parser/extract.lua but one thing that I still don't understand is the role of the returned value of the ingest function. Is there some documentation about it?

Sorry for the long post 😭

[stevearc]

For your first question: is there a better way to get the gcc parser? I would say yes. The way you're fetching and then mutating the problem matcher right now is quite brittle. It makes more sense to just copy the regular expression it's using and use that directly in your own parser. Here's where it's defined:

overseer.nvim/lua/overseer/template/vscode/problem_matcher.lua

Lines 264 to 274 in 16ac26a

	["$gcc"] = {
	fileLocation = { "autoDetect", "${cwd}" },
	pattern = {
	regexp = "^(.*?) =(\\d+):(\\d*):?\\s+(?:fatal\\s+)?(warning|error):\\s+(.*)$",
	vim_regexp = "\\v^([^:]*):(\\d+):(\\d*):?\\s+%(fatal\\s+)?(warning|error):\\s+(.*)$",
	file = 1,
	line = 2,
	column = 3,
	severity = 4,
	message = 5,
	},

Using it directly would look like this:

components = {
  "defaults",
  {
    "on_output_parse",
    parser = {
      diagnostics = {
        {
          "extract",
          { append = false, regex = true },
          "\\v^([^:]*):(\\d+):(\\d*):?\\s+%(fatal\\s+)?(warning|error):\\s+(.*)$",
          "filename",
          "lnum",
          "col",
          "type",
          "text",
        },
        { "extract_multiline", "%s+%d*%s*|(%s+.*)", "text" },
      },
    },
  },
}

For your second question: how can you count the ~ characters to set the end_col? This is something that you can do with a postprocess function on the item. Unfortunately the extract_multiline leaf doesn't have a postprocess argument (probably just an oversight on my part), but you can set append = false and add an append leaf which does. You can do that like so:

components = {
  "defaults",
  {
    "on_output_parse",
    parser = {
      diagnostics = {
        {
          "extract",
          { append = false, regex = true },
          "\\v^([^:]*):(\\d+):(\\d*):?\\s+%(fatal\\s+)?(warning|error):\\s+(.*)$",
          "filename",
          "lnum",
          "col",
          "type",
          "text",
        },
        { "extract_multiline", { append = false }, "%s+%d*%s*|(%s+.*)", "text" },
        {
          "append",
          {
            postprocess = function(item)
              local lines = vim.split(item.text, "\n", { plain = true, trimempty = true })
              local last_line = lines[#lines]
              -- Set the end_col to be the last character of the last line we parsed
              item.end_col = string.len(last_line)
            end,
          },
        },
      },
    },
  },
}

Also note that the sequence in your example was redundant, since a list gets wrapped in a sequence anyway.

One last point: if you haven't seen or used the parser debugger yet (documented here) I would highly recommend it. It makes writing and debugging parsers infinitely easier.

[vheon]

is there a better way to get the gcc parser? I would say yes. The way you're fetching and then mutating the problem matcher right now is quite brittle. It makes more sense to just copy the regular expression it's using and use that directly in your own parser

The first instinct was to do it like this so that I wouldn't have to copy the regex and in case that was updated I would get it updated for free but I admit that given the result it doesn't seems much of a win the way I did it.

I couldn't find a way to do it reading the documentation (or maybe I missed the needed piece) so I thought of writing a parser that would do this custom thing mimicking the implementation of https://github.com/stevearc/overseer.nvim/blob/master/lua/overseer/parser/extract.lua but one thing that I still don't understand is the role of the returned value of the ingest function. Is there some documentation about it?

Any info on this last question?

[stevearc]

There's no documentation for writing your own nodes in the behavior tree parser. It's not a public API and I don't expect people to need to write their own nodes. Is there something you're still trying to accomplish that can't be done with the existing nodes?

[vheon]

I guess that you demonstrated that it can be done composing what it is available. Its just that it would probably be as readable or more if it was a single parser, that's all.

[wjakob]

@vheon Did you ever get this to work, and would you be able to share a complete example in this case? I'm also trying to hook up Overseer.nvim and Trouble.nvim to have a nice navigation of compilation failures/warnings in a C++-based project.

[vheon]

@wjakob these are the components I use in the C++ project I work on daily:

                            components = {
                                { "display_duration", detail_level = 2 },
                                "on_output_summarize",
                                "on_exit_set_status",
                                "on_complete_notify",
                                {
                                    "on_output_parse",
                                    parser = {
                                        diagnostics = {
                                            {
                                                "extract",
                                                { append = false, regex = true },
                                                "\\v^([^:]*):(\\d+):(\\d*):?\\s+%(fatal\\s+)?(warning|error):\\s+(.*)$",
                                                "filename",
                                                "lnum",
                                                "col",
                                                "type",
                                                "text",
                                            },
                                            { "extract_multiline", "%s+%d*%s*|(%s+.*)", "text" },
                                        }
                                    }
                                },
                                { "on_result_diagnostics", remove_on_restart = true },
                                "unique"
                            },

This has its own problem, for example it doesn't capture link errors and for template errors it doesn't capture the location where it is used but the location of the template, but it is a starting point :)