Reading large JSON databases is slow

[Hylen]

Hi,

I've found that irony-cdb-autosetup-compile-options can be really slow when reading large JSON compilation databases. I noticed this when using irony-mode on llvm itself. Every time I open a new source file there is a ~5s delay.

Based on my attempts at profiling, it seems most of the time is spent in irony-cdb-json--transform-command. Time is also spent in json-read-file (emacs built-in). I am no elisp expert, but as I understand, this only reads and postprocess the database a little. The database file is ~10000 lines, 2 MB, so I don't understand where the 5s come form. Maybe it is the communication to the server? The elisp profiler may not see this time.

Anyway, I suggest moving the database reading to the server. I might also be willing to do this, if you agree. It would be a good way to read up libclang and one of my favorite emacs extensions :)

/Karl

Profiler output:

- ...                                                            3387  92%
 - irony-cdb--autodetect-compile-options                         2942  80%
  - catch                                                        2942  80%
   - let                                                         2942  80%
    - while                                                      2942  80%
     - let                                                       2942  80%
      - funcall                                                  2942  80%
       - irony-cdb-json                                          2942  80%
        - cond                                                   2942  80%
         - irony-cdb-json--get-compile-options                   2942  80%
          - let                                                  2942  80%
           - if                                                  2942  80%
            - progn                                              2942  80%
             - progn                                             2942  80%
              - let                                              2942  80%
               - irony-cdb-json--load-db                         2942  80%
                - delq                                           2942  80%
                 - mapcar                                        2939  80%
                  + irony-cdb-json--transform-compile-command               1958  53%
                  + json-read-file                                981  26%
   Automatic GC                                                   445  12%
+ command-execute                                                 254   6%
+ timer-event-handler                                              10   0%

[Sarcasm]

Hi,

Yeah, I'm aware of the issue and I have some thoughts on that, I'm still not 100% sure about what's the proper solution.

Based on my attempts at profiling, it seems most of the time is spent in irony-cdb-json--transform-command.

I'm not surprised here, one of the function that may be quite inefficient is irony--split-command-line.

Time is also spent in json-read-file (emacs built-in). I am no elisp expert, but as I understand, this only reads and postprocess the database a little.

I think you understand right. The parsing method is not very efficient but I think it may be difficult to get one that scales well in Elisp (I may be wrong).

The database file is ~10000 lines, 2 MB, so I don't understand where the 5s come form. Maybe it is the communication to the server?

That is not the server, for the compilation database everything happens on the Elisp side.

Anyway, I suggest moving the database reading to the server. I might also be willing to do this, if you agree. It would be a good way to read up libclang and one of my favorite emacs extensions :)

That would be a welcomed change. One thing to take care of is to be able to deduce the compile options if the entry is not directly specified in clang_CompilationDatabase_getCompileCommands(). This logic is what the undocumented function irony-cdb-json--guess-flags does. Let me know if you want me to clear that up.

Right now I'm trying to get my head around about a somewhat clean API for the communication with irony-server, so implementing a new commands do not require to copy/pasta the same code/logic every time. So if you happen to implement an irony-server command, try to make things very simple so I can update it easily once I'm satisfied with the irony-server command.

I'm not sure I'm any clear but I would be glad if you work on this, try to do something simple first and I will comment on it.

[Hylen]

Hi again,

I skimmed through the elisp and the server code, and I have some basic ideas. As of now, the whole database is read, processed and then only the options corresponding to the current file are filtered out. Maybe the database could be stored in memory?

That would be a welcomed change. One thing to take care of is to be able to deduce the compile options if the entry is not directly specified in clang_CompilationDatabase_getCompileCommands(). This logic is what the undocumented function irony-cdb-json--guess-flags does. Let me know if you want me to clear that up.

I think I kind of understand what happens here. I will look more closely at elisp later. Didn't find the explanation you referred to, but the function is interesting. If I understand correctly, clang_CompilationDatabase_fromDirectory can be used to read the database on the server. I will experiment with it when I got the time.

Right now I'm trying to get my head around about a somewhat clean API for the communication with irony-server, so implementing a new commands do not require to copy/pasta the same code/logic every time. So if you happen to implement an irony-server command, try to make things very simple so I can update it easily once I'm satisfied with the irony-server command.

Understood, I will not touch the server API too much. I guess that I will need to remove passing of the compile options read from the database, but nothing more than that. I will also keep the user defined options in irony-additional-clang-options.

/Karl

[Sarcasm]

I skimmed through the elisp and the server code, and I have some basic ideas. As of now, the whole database is read, processed and then only the options corresponding to the current file are filtered out. Maybe the database could be stored in memory?

Yes, caching the database should help but I would like to wait before doing it. The parsing should be fast anyway, not only starting from the second time.

If I understand correctly, clang_CompilationDatabase_fromDirectory can be used to read the database on the server.

Yes that's the one we want to use.

Understood, I will not touch the server API too much. I guess that I will need to remove passing of the compile options read from the database, but nothing more than that. I will also keep the user defined options in irony-additional-clang-options.

I don't think that's what we want. It is important that Emacs has access to these flags because they may be used for other things than irony-server's commands.

I think the easiest thing to start would be to have a command in irony-server compile-options FILE BUILD-DIR or something like that, that returns the compile options for the current file (using the same algorithms as the elisp one but on the server side).

Then this function should be called in place of irony-cdb-json--get-compile-options. Right now there is not much support for synchronous call in irony--send-request so that may be another problem for you. The refactoring I'm currently doing on the irony-server side should probably take into account the need for synchronous requests.

There will be room for further improvement but that may help already. Then we can improve on this (e.g: caching the compilation database results if the file modification time didn't change).

[Hylen]

I don't think that's what we want. It is important that Emacs has access to these flags because they may be used for other things than irony-server's commands.

I think the easiest thing to start would be to have a command in irony-server compile-options FILE BUILD-DIR or something like that, that returns the compile options for the current file (using the same algorithms as the elisp one but on the server side).

Then this function should be called in place of irony-cdb-json--get-compile-options. Right now there is not much support for synchronous call in irony--send-request so that may be another problem for you. The refactoring I'm currently doing on the irony-server side should probably take into account the need for synchronous requests.

I were not aware of this requirement. Then let's for now create a second executable that can be called to read database synchronously. When the server api supports it, it can be moved inside irony-server. This way I will not have to touch anything in the server API, that you are working on.

/Karl

[Sarcasm]

You can implement the command in the irony-server executable but call irony-server synchronously (e.g: with call-process):

~/.emacs.d/irony/bin/irony-server get-compile-options <build-dir>

[Hylen]

Is it ok to include boost as a dependency? It might be the easiest way to make routines such as expand-file-name portable on the server side.

I also have some thoughts on the server API. According to the section on filter functions in the elisp manual, the filter functions can read only when emacs waits. If we want to simulate a synchronous call to irony-server, we might use the asynchronous methods currently used in combination with sit-for or sleep-for. This way, the options can be fetched from the sever and the server and need not be passed the options from elisp. We would have to write the .clang_complete stuff on the server as well, but that is a minor problem. What do you think?

[Sarcasm]

Is it ok to include boost as a dependency?

Nope, I understand the issue but I really want irony to be self-contained, apart from the obvious need to have libclang. There is an alternative solution other than to replicate the full logic of the elisp code into C++. Since it's parsing the JSON and parsing the command line that takes time, it would be possible to "compile" the compilation database into an s-expr, just doing a server implementation of irony-cdb-json--load-db.

I also have some thoughts on the server API. According to the section on filter functions in the elisp manual, the filter functions can read only when emacs waits. If we want to simulate a synchronous call to irony-server, we might use the asynchronous methods currently used in combination with sit-for or sleep-for.

That is not the proper way to do waiting on asynchronous processes. One should use accept-process-output. I actually have a comment on irony--parse-buffer-async which reflect the current thought I have on synchronous methods.

This way, the options can be fetched from the sever and the server and need not be passed the options from elisp. We would have to write the .clang_complete stuff on the server as well, but that is a minor problem. What do you think?

We don't want to do this. We really want Emacs to "master" the command line options so that it can interface with:

• Emacs' built-in modules, like ff-find-other-file (e.g: irony-cdb could set cc-search-directories)
• external tools, e.g: call a compiler to generate assembly of the current file/function, like disaster
• or to interface with some other Clang tools (clang-rename, clang-tidy, clang-modernize, ...).

For now I think it's better to optimize the parsing via irony-server but it shouldn't have too much power and state if possible.

[Hylen]

Nope, I understand the issue but I really want irony to be self-contained, apart from the obvious need to have libclang. There is an alternative solution other than to replicate the full logic of the elisp code into C++. Since it's parsing the JSON and parsing the command line that takes time, it would be possible to "compile" the compilation database into an s-expr, just doing a server implementation of irony-cdb-json--load-db.

Even if we only do irony-cdb-json--load-db on the server, it still needs a portable equivalent to expand-file-name. Do you suggest we do this from scratch? If we really want it to be portable, I think this could be kind of involved. Or do you suggest using the preprocessor to call different utility functions on different systems, like realpath?

That is not the proper way to do waiting on asynchronous processes. One should use accept-process-output. I actually have a comment on irony--parse-buffer-async which reflect the current thought I have on synchronous methods.

Alright, hadn't seen that one. Then synchronous communication shouldn't be that hard.

We don't want to do this. We really want Emacs to "master" the command line options so that it can interface with:

Emacs' built-in modules, like ff-find-other-file (e.g: irony-cdb could set cc-search-directories)
external tools, e.g: call a compiler to generate assembly of the current file/function, like disaster
or to interface with some other Clang tools (clang-rename, clang-tidy, clang-modernize, ...).
For now I think it's better to optimize the parsing via irony-server but it shouldn't have too much power and state if possible.

Alright, got it.

[Sarcasm]

Even if we only do irony-cdb-json--load-db on the server, it still needs a portable equivalent to expand-file-name. Do you suggest we do this from scratch? If we really want it to be portable, I think this could be kind of involved. Or do you suggest using the preprocessor to call different utility functions on different systems, like realpath?

I think we could start by just having a function that says whether or not "file" is relative, and if that's so prefix it with the directory. I think we will be pretty good with this. Maybe, if we do only irony-cdb-json--load-db, we can still do the expand-file-name on the Elisp side. I think what is important to have is the command line parsing to be done in C++.

Given:

[
  { "directory": "/home/user/llvm/build",
    "command": "/usr/bin/clang++ -Irelative -DSOMEDEF=\"With spaces, quotes and \\-es.\" -c -o file.o file.cc",
    "file": "file.cc" },
  ...
]

We could return:

("file.cpp" "/home/user/llvm/build" '("/usr/bin/clang++" "-Irelative" "-DSOMEDEF=\"With spaces, quotes and \\-es.\"" "-c" "-o" "file.o" "file.cc"))

Hopefully more work can be done but that's a start.

Alright, hadn't seen that one. Then synchronous communication shouldn't be that hard.

Nope, it's not too difficult, irony used to be synchronous a long time ago but asynchronous was better in the end. And now I think both are useful. :)

[Hylen]

I think we could start by just having a function that says whether or not "file" is relative, and if that's so prefix it with the directory. I think we will be pretty good with this. Maybe, if we do only irony-cdb-json--load-db, we can still do the expand-file-name on the Elisp side. I think what is important to have is the command line parsing to be done in C++.

What do you mean with command line parsing exactly? irony-cdb-json--transform-compile-command? I think the most important thing is to single out one compile command on the server. I think for instance that the guess-flags logic have to be implemented on the server side.

I have almost completely written the code in irony-cdb-json--get-compile-options, and it should work for UNIX based systems. Only thing left is the guess-flags routine. When I started writing this, I noticed a problem. libclang doesn't have a way of getting the target file name from c CXCompileCommand. I would be glad for workaround advice here. Only solution I see is to implement the parsing too (since I guess including a JSON parsing library isn't an option).

[Hylen]

How do you feel about switching from libclang to libtooling? I think this would solve the problem above and make the code much simpler. Another upside is that it would be safer and nicer to use the C++ API. I guess the downside is that it is less stable.

[Sarcasm]

I have almost completely written the code in irony-cdb-json--get-compile-options, and it should work for UNIX based systems.

Are you using some kind of syscalls for it to be specific to Linux or is is the path parsing methods that are Linux-specific? I would like to avoid relying on syscall which may be slow on some file system when we should be able to do something quite cheap just with path parsing.

I think for instance that the guess-flags logic have to be implemented on the server side.

Maybe that is a good idea but if it's not slow in Elisp and can be cached easily then it is no big deal. It's better if it's done in C++ but if the overall code is way more complex for no visible benefit that may not be worth the maintenance burden.

When I started writing this, I noticed a problem. libclang doesn't have a way of getting the target file name from c CXCompileCommand. I would be glad for workaround advice here. Only solution I see is to implement the parsing too (since I guess including a JSON parsing library isn't an option).

Damn, that's unfortunate. :-( I was really glad that I didn't have to add a JSON parsing library by seeing the compilation database support but I was wrong apparently. A self-hosted JSON parsing library is an option if libclang doesn't provide this information, something like rapidjson.

How do you feel about switching from libclang to libtooling? I think this would solve the problem above and make the code much simpler. Another upside is that it would be safer and nicer to use the C++ API. I guess the downside is that it is less stable.

I prefer to stay with libclang whose exact purpose is to be used by IDE. I like the fact the API is stable, irony-mode support all kind of versions and I think it is a good thing. I wouldn't be against having another program that uses LibTooling but that would be something optional I think.

[Hylen]

Are you using some kind of syscalls for it to be specific to Linux or is is the path parsing methods that are Linux-specific? I would like to avoid relying on syscall which may be slow on some file system when we should be able to do something quite cheap just with path parsing.

Just the parsing, and this should be quite easy to fix.

Maybe that is a good idea but if it's not slow in Elisp and can be cached easily then it is no big deal. It's better if it's done in C++ but if the overall code is way more complex for no visible benefit that may not be worth the maintenance burden.

Damn, that's unfortunate. :-( I was really glad that I didn't have to add a JSON parsing library by seeing the compilation database support but I was wrong apparently. A self-hosted JSON parsing library is an option if libclang doesn't provide this information, something like rapidjson.

It would be nice to have it on the server, since it includes a couple of passes over the whole database. But at the same time, I guess the common case is to get the flags directly without guessing. So maybe a first attempt should be something like:

1. Implement the exact logic on the server (irony-cdb-json--load-db + irony-cdb-json--exact-flags)
2. If this fails, load the database again in elisp and do the guessing

Using this method, if you have your file in the database, the loading of the commands should be fast. No dependencies need to be added.

I prefer to stay with libclang whose exact purpose is to be used by IDE. I like the fact the API is stable, irony-mode support all kind of versions and I think it is a good thing. I wouldn't be against having another program that uses LibTooling but that would be something optional I think.

I understand this reason. It is up to you which way we take. For now, I focus on the solution above.

[Sarcasm]

I agree with what you are saying, I think it's a first step in the good direction. 👍

[Hylen]

Hi,

Check out the server-database branch in my fork. I've added the the exact flags logic on the server, and made a way of communicating synchronously with irony-server. I still have some stuff I want to do before I send a pull request (documented using TODO comments, and some in my head) but I wanted to here your thoughts before working more on it.

Regards,
Karl