Crash in libclang: ASTTypeWriter::VisitTagType() and CompilerInstance::~CompilerInstance()

[Sarcasm]

@DanSchoppe: I'm creating a dedicated issue for: #137 (comment)

Installation: I installed company, company-irony, and irony-mode packages from Melpa. I updated my init.el to include the suggested entries for the three packages, exactly how LefterisJP listed in his first post. I built and installed irony server pointing to my homebrew installation of llvm 3.5.1 by adding the -DCMAKE_PREFIX_PATH=[path/to/llvm/3.5.1] to the cmake args.

So you are using LLVM 3.5.1, what about your OS?

Project: Makefile project, so I used bear to generate a JSON database containing compile flags. The compile_commands.json file looks like it has a list of flags for each object as I would expect. Just like LefterisJP, I am successfully using this JSON database with RTags.

Since it's not using the CMake generator, I would be interested in seeing the flag in use. For example, it seems that -Qunused-arguments doesn't play nice with libclang, see Sarcasm/company-irony#8

Printing the output of M-x irony-cdb-menu RET would be helpful.

When I run irony-completion-at-point-async in a project file, I get back "Wrong type argument: sequencep, libclang:" After toggling on debug on error I get a backtrace with the same contents as LefterisJP listed earlier.

You got a libclang crash, that's how it is printed for now, this is a known issue, ideally I should intercept this, see #157.

Here's where I start to get suspicious... When I inspect irony--compile-options (after running irony-completion-at-point-async), only the first entry from the compile_commands.json file is listed in the variable.

I believe that's is expected, irony--compile-options is a buffer-local variable, that contains the compile options in use for the current buffer, it is not the full JSON compilation database (this one doesn't stay in memory).

I also tested the irony-server in a bash terminal (not sure if this is relevant or helpful):

./irony-server -d complete [path/to/source/file.cpp] 100 7 RET RET

execute: Command{action=Command::Complete, file='[path/to/source/file.cpp]', line=100, column=7, flags=[], unsavedFiles.count=0, opt=off}
Assertion failed: (!T->isBeingDefined() && "Cannot serialize in the middle of a type definition"), function VisitTagType, file ASTWriter.cpp, line 277.
libclang: crash detected during reparsing
Assertion failed: (OutputFiles.empty() && "Still output files in flight?"), function ~CompilerInstance, file CompilerInstance.cpp, line 62. Abort trap: 6

Testing in a bash terminal is very helpful:

It would be nice if you could:

• provide the compile options. this is possible if you do:
  • ./irony-server -d complete [path/to/source/file.cpp] 100 7 RET
  • -std=c++11 -I<path/to/include> <other compile options...> RET
• tell me if it happens with every file, even the most trivial one or if you can reduce the file you have an issue with

The crash seems to be in libclang but maybe irony isn't using the API properly somewhere.

[DanSchoppe]

Hi Sarcasm,

Thanks for the helpful response. Sorry for originally posting in the wrong thread.

So you are using LLVM 3.5.1, what about your OS?

OS X 10.10.3

Printing the output of M-x irony-cdb-menu RET would be helpful.

Compilation Database: irony-cdb-json
  Working Directory: [path/to/project]
  Compile Options:   -std=c++1y -stdlib=libc++ -Wall -MMD -MP -Wno-deprecated-declarations -I./src -isystem /Users/danschoppe/local/include/ -isystem /Users/danschoppe/local/include/libxml2 -isystem ./external -isystem ./build/msgpack/include -isystem /usr/local/include -O0 -fno-inline -g -Ibuild/debug -DBUILD_VERSION=v1.0.0 -DBUILD_TIME=2015-04-13 23:59:15-0500 -isystem ./OSX/deps -mmacosx-version-min=10.8 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk

I believe that's is expected, irony--compile-options is a buffer-local variable, that contains the compile options in use for the current buffer, it is not the full JSON compilation database (this one doesn't stay in memory).
That makes sense, then. The buffer-local variable does match the JSON entry for the file I am editing.

Running the irony-server with compile options results in this error:

Assertion failed: (!T->isBeingDefined() && "Cannot serialize in the middle of a type definition"), function VisitTagType, file ASTWriter.cpp, line 277.
libclang: crash detected during reparsing
Assertion failed: (OutputFiles.empty() && "Still output files in flight?"), function ~CompilerInstance, file CompilerInstance.cpp, line 62.
Abort trap: 6

tell me if it happens with every file, even the most trivial one or if you can reduce the file you have an issue with

I'll have to do some testing. Can I test with the server binary and line 1 column 1 on a bunch of source files (with their flags), or do I actually need to find a reasonable line and column in the file? I could also build a quick test project to test on something simple.

Thanks,
Dan

[Sarcasm]

Okay,

I think some flags may be responsible for this:

• -MMD and/or -MP: like -Qunused-arguments, irony should probably filter-out these ones
• -DBUILD_VERSION=v1.0.0 if v1.0.0 is supposed to be a string
• -DBUILD_TIME=2015-04-13 23:59:15-0500 The spaces may not be understood by irony, this is a known issue, see deal with compiler parameters that have spaces in them #178

I'll have to do some testing. Can I test with the server binary and line 1 column 1 on a bunch of source files (with their flags), or do I actually need to find a reasonable line and column in the file? I could also build a quick test project to test on something simple.

You can use parse instead of complete, so you don't have to think about lines and columns.

[DanSchoppe]

I tried autocomplete on a simpler file that uses all the same compiler flags (including -MMD, -MP, -DBUILD_VERSION=string, -DEBUILD_TIME=stringWithSpaces) and everything seems to work. Irony-server parse works great on this file (at least it takes a second or two then reports "t \n\n ;;EOT". irony-server complete looked like it was working as well, returning reasonable suggestions.

In Emacs I can put my cursor after the dereference of a custom class pointer inside the simpler file, run M-x irony-completion-at-point-async RET and get "Complete, but not unique". When I run M-x company-complete, I get the dropdown list of all the public member variables and member functions, just like I'd expect! The only weird thing is that after I get the successful autocomplete, if I run M-x irony-cdb-menu, it responds "No compilation database in use. [q] to quit"

I suppose that narrows down the problem to libclang parsing the first file. That seems strange since it builds and runs fine. The next step to identifying the root cause is to probably start commenting out portions of the problem file.

Thanks,
Dan

[Sarcasm]

When I run M-x company-complete, I get the dropdown list of all the public member variables and member functions, just like I'd expect!

To be extra-sure that you are using company-irony and not company-clang, or company-capf you should call company-irony explicitly: M-x company-irony RET.

The only weird thing is that after I get the successful autocomplete, if I run M-x irony-cdb-menu, it responds "No compilation database in use. [q] to quit"

Before the successful complete it wasn't empty? I would be surprised if the compile option changes for no apparent reasons. This is not something that is "smartly" refreshed or anything, the flags are set when irony-cdb-autosetup-compile-options is called, this is done in irony-mode-hook if you follow the standard configuration steps and that is all.

I suppose that narrows down the problem to libclang parsing the first file. That seems strange since it builds and runs fine. The next step to identifying the root cause is to probably start commenting out portions of the problem file.

Yep you can try that.

I was wondering if you could print the value of the variable: C-h v irony--compile-options RET.
Maybe -DBUILD_TIME or something like that isn't escaped properly.

For testing purpose you can also edit this variable directly to remove the suspicious compile options.

[DanSchoppe]

To be extra-sure that you are using company-irony and not company-clang, or company-capf you should call company-irony explicitly: M-x company-irony RET.

Yep, if I call M-x company-irony RET on a simple file I get proper autocompletion.

The only weird thing is that after I get the successful autocomplete, if I run M-x irony-cdb-menu, it responds "No compilation database in use. [q] to quit"

Before the successful complete it wasn't empty? I would be surprised if the compile option changes for no apparent reasons. This is not something that is "smartly" refreshed or anything, the flags are set when irony-cdb-autosetup-compile-options is called, this is done in irony-mode-hook if you follow the standard configuration steps and that is all.

It was empty before and after calling irony async. I guess I'm surprised that I'm getting valid auto completions if the irony-cdb-menu says "No compilation database in use." Apparently this message does not indicate a problem?

irony--compile-options is a variable defined in `irony.el'.
Its value is
("-std=c++1y" "-stdlib=libc++" "-Wall" "-MMD" "-MP" "-Wno-deprecated-declarations" "-I./src" "-isystem" "/Users/danschoppe/local/include/" "-isystem" "/Users/danschoppe/local/include/libxml2" "-isystem" "./external" "-isystem" "./build/msgpack/include" "-isystem" "/usr/local/include" "-O0" "-fno-inline" "-g" "-Ibuild/debug" "-DBUILD_VERSION=v1.0.0" "-DBUILD_TIME=2015-04-13" "23:59:15-0500" "-isystem" "./OSX/deps" "-mmacosx-version-min=10.8" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk")
Local in buffer [file].cpp; global value is nil
  Automatically becomes buffer-local when set.
Documentation:
Compile options for the current file.
The compile options used by the compiler to build the current
buffer file.

[Sarcasm]

Okay, as you can see this is a bug in the compile options:

You have:

"-DBUILD_VERSION=v1.0.0" "-DBUILD_TIME=2015-04-13" "23:59:15-0500"

I think it should be:

"-DBUILD_VERSION=\"v1.0.0\"" "-DBUILD_TIME=\"2015-04-1323:59:15-0500\""

Type M-:, and evaluate the following:

(setq irony--compile-options '("-std=c++1y" "-stdlib=libc++" "-Wall" "-Wno-deprecated-declarations" "-I./src" "-isystem" "/Users/danschoppe/local/include/" "-isystem" "/Users/danschoppe/local/include/libxml2" "-isystem" "./external" "-isystem" "./build/msgpack/include" "-isystem" "/usr/local/include" "-O0" "-fno-inline" "-g" "-Ibuild/debug" "-DBUILD_VERSION=\"v1.0.0\"" "-DBUILD_TIME=\"2015-04-13 23:59:15-0500\"" "-isystem" "./OSX/deps" "-mmacosx-version-min=10.8" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk")

If it doesn't work, and if you know if you can compile without -DBUILD_VERSION and -DBUILD_TIME, you can try to remove these one as well, maybe I failed the string escaping.

It was empty before and after calling irony async. I guess I'm surprised that I'm getting valid auto completions if the irony-cdb-menu says "No compilation database in use." Apparently this message does not indicate a problem?

Without compile options libclang is able to figure some things out but since it cannot find header files it will not complete structures from header files, it will not be able to compute a preamble so things may be sluggish, ...It still manage to do some things but that's not the best. "No compilation database in use" may be a valid message in a new buffer for example, the message isn't necessarily a sign of error but in your case it is not right.

[DanSchoppe]

I tried removing the -DBUILD_VERSION and -DBUILD_TIME from my project but was running into the same issue. I also did some irony-server parse testing with and without the -DBUILD_* flags but the behavior did not change. Some files parse fine, others hit this libclang crash. It seems like using irony-server parse is the most controlled environment I have to track down the issue.

Is there a way to use irony-server with a different version of llvm, or do I need to recompile the server? I'm curious to try it with 3.6 instead of 3.5.1.

[Sarcasm]

You have to recompile but it shouldn't be too difficult since you already know about CMAKE_PREFIX_PATH.

[DanSchoppe]

Sarcasm,

I brew installed llvm 3.6.0 and recompiled irony-server. I needed to set some additional flags in order for the installation to find the include dir and lib (I suspect because llvm36 uses command "llvm-config-3.6" instead of "llvm-config"):

-DCMAKE_PREFIX_PATH\=/Users/danschoppe/local/Cellar/llvm36/3.6.0/
-DLIBCLANG_INCLUDE_DIR\=/Users/danschoppe/local/Cellar/llvm36/3.6.0/lib/llvm-3.6/include
-DLIBCLANG_LIBRARY\=/Users/danschoppe/local/Cellar/llvm36/3.6.0/lib/llvm-3.6/lib/libclang.dylib

I used the new irony-server to try parsing some files that previously failed to parse. It worked, even with the -DBUILD_VERSION and -DBUILD_TIME flags! But to reduce variables, I did recompile my application without the -DBUILD_VERSION and -DBUILD_TIME flags.

So I hopped back to Emacs and tried M-x company-irony in a few spots. In some files it was working great, and in other files It was still reporting "Cannot complete at point". M-x irony-completion-at-point-async in the problematic files still reports "Wrong type argument: sequencep, libclang:" If I run irony-server -d complete [file] [line] [column] on a problem file, some errors are output before it hits a seg fault:

[file]:65:14: error: unknown type name 'optional'
[file]:65:22: error: expected ')'
[file]:62:13: note: to match this '('
[several more errors]
[file]:79:8: error: exception specification in declaration does not match previous declaration
Segmentation fault: 11

Some thoughts:

• I didn't do any additional configuration of irony after recompiling the irony-server or my application. I'm not sure if there are things I need to do.
• M-x irony-cdb-menu looks good for all files I tested. It shows it's using a irony-cdb-json database in the correct working directory and with the correct compile options:

  
  -std=c++1y -stdlib=libc++ -Wall -MMD -MP -Wno-deprecated-declarations -I./src -isystem /Users/danschoppe/local/include/ -isystem /Users/danschoppe/local/include/libxml2 -isystem ./external -isystem ./build/msgpack/include -isystem /usr/local/include -O0 -fno-inline -g -Ibuild/debug -isystem ./OSX/deps -mmacosx-version-min=10.8 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk
  

• I'm using llvm 3.5.1 to compile, but llvm 3.6.0 for irony. Is this an issue?
• Since llvm 3.5.1 is the default installed version when I run brew install llvm, the llvm-config command is part of the 3.5.1 package and the llvm-config-3.6 command is for the 3.6.0 package. It didn't seem like the irony-server handled this gracefully when I compiled it (based on the fact that I had to link the LIBCLANG_INCLUDE_DIR and LIBCLANG_LIBRARY), so I'm hoping that there aren't other places that irony uses "default" commands instead of appended with "-3.6".

[DanSchoppe]

Update: I think it's working! It seems to be working after running your suggested M-: command:

(setq irony--compile-options '("-std=c++1y" "-stdlib=libc++" "-Wall" "-Wno-deprecated-declarations" "-I./src" "-isystem" "/Users/danschoppe/local/include/" "-isystem" "/Users/danschoppe/local/include/libxml2" "-isystem" "./external" "-isystem" "./build/msgpack/include" "-isystem" "/usr/local/include" "-O0" "-fno-inline" "-g" "-Ibuild/debug" "-DBUILD_VERSION=\"v1.0.0\"" "-DBUILD_TIME=\"2015-04-13 23:59:15-0500\"" "-isystem" "./OSX/deps" "-mmacosx-version-min=10.8" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk"))

So what do you make of all this?

Thanks for the help,
Dan

[Sarcasm]

I brew installed llvm 3.6.0 and recompiled irony-server. I needed to set some additional flags in order for the installation to find the include dir and lib (I suspect because llvm36 uses command "llvm-config-3.6" instead of "llvm-config"):

I'm not sure but I think that you may have to use a different prefix path.

Either:

-DCMAKE_PREFIX_PATH=/Users/danschoppe/local/Cellar/llvm36/

Or:

-DCMAKE_PREFIX_PATH=/Users/danschoppe/local/Cellar/llvm36/3.6.0/lib/llvm-3.6/

If you have ldd or similar on OS X, I would be curious to see the libclang that is picked up at runtime.

To verify on Linux, you would do:

$ ldd ~/.emacs.d/irony/bin/irony-server | grep libclang
        libclang.so => /usr/lib/libclang.so (0x00007fb60d7c4000)

In some case it may be necessary to tell CMake to set the RPATH so the proper lib is found, and not the system wide one (see Bottom of FAQ: https://github.com/Sarcasm/irony-mode#faq).

Can you make sure the built-in headers are found when you build with llvm 3.6:

You should have something like this:

$ cmake /path/to/irony-mode
...
-- Detecting libclang builtin headers directory
-- Detecting libclang builtin headers directory -- success
...

Update: I think it's working! It seems to be working after running your suggested M-: command:

(setq irony--compile-options '("-std=c++1y" "-stdlib=libc++" "-Wall" "-Wno-deprecated-declarations" "-I./src" "-isystem" "/Users/danschoppe/local/include/" "-isystem" "/Users/danschoppe/local/include/libxml2" "-isystem" "./external" "-isystem" "./build/msgpack/include" "-isystem" "/usr/local/include" "-O0" "-fno-inline" "-g" "-Ibuild/debug" "-DBUILD_VERSION="v1.0.0"" "-DBUILD_TIME="2015-04-13 23:59:15-0500"" "-isystem" "./OSX/deps" "-mmacosx-version-min=10.8" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk"))
So what do you make of all this?

I don't know, don't understand everything.

Can you use the CDB without manual modification, and print the content of irony--compile-options so that we can compare the differences in command line?

Then I guess, one would have to try each of the differences to find out which one affects the result.