Build using compile_commands.json to automatically assign language for discovered sources

[i-garrison]

There are issues using current CDT (with original Indexer) with GDB The GNU Project Debugger sources. I understand there is a new cdt-lsp to try - unfortunately that is not as usable for me at the moment so maybe we can improve original implementation too.

Currently GDB The GNU Project Debugger sources are a mix of C and C++ language code where lots (about 500) of files have .c extension but are in fact already transitioned to C++ and are compiled using C++ compiler (here, g++). The other lots (also about 500) of .c files are remaining C code and that of dependencies which are also in C language and are compiled using C compiler (here, gcc). The major issues I see are:

1. I'm unable to configure CDT Indexer properly since I only know about file type mappings by file extension.
   With that I can either configure per-project File Types entry*.c -> C++ source file which breaks Indexer for the other half because lots of C constructs are not valid C++, or do nothing and C++ half of files is not parsed properly because C Indexer parser is used.

2. I'm using Compilation Database Parser with GDB sources and it helps a lot but at the moment compilation database parser ask project to select language for discovered source file and uses that information to populate language-specific entry for macros etc..
   This does not work for GDB source since about half of .c files are C++ and due to first issue discovered macros end up in the wrong language-specific bin with 50% chance.

Ideally I'd like Compilation Database Parser to use discovered language type from compilation database and configure language for discovered source files in project for me. It already knows which compiler (e.g. g++ or gcc) is used for each of compiled files so this should be possible. This probably requires adding a mapping selection of compiler to language in compilation database parser options but that should not be hard part.

[ewaterlander]

Renaming the C++ files to a proper C++ extension would help every editor or IDE.

Have you tried Core Build Makefile instead of Managed Build Makefile? Core Build Makefile is using the console output to discover compilation info.

[i-garrison]

Renaming the C++ files to a proper C++ extension would help every editor or IDE.

No of course this is not what you normally do while working with codebase to explore it, and for GDB it is also impractical due to the numer of files affected.

Have you tried Core Build Makefile instead of Managed Build Makefile? Core Build Makefile is using the console output to discover compilation info.

Not yet but will do now. Side question - turned out that according to recent documentation Core Build relies on Launch Bar feature to enable build configuration. I was successfully updating my Eclipse CDT installation via Check for Updates through several releases (guess about 3 years?) and this feature is missing in mine so I will be installing it by hand.

[i-garrison]

Hi @ewaterlander I did a few more experiments with Core Build Makefile (some other thoughts are in #1332) and unfortunately I do not think that fits. GDB by default is built in quiet mode so you get
CXX gdb.o
then I set build command to make V=1 and it is now
g++ -x c++ -I. -I../../gdb -I../../gdb/config ... ../../gdb/gdb.c
but here g++ is the driver, and language is -x c++ so I believe it will miss language in the end.
Even though it currently could be fortunate and parse g++ in the above command as C++ language indicator, it does not currently solve my issue - after all of the above and rebuilding GDB from scratch gdb/gdb.c still has C Source File type and has a lots of parser errors in it.

Side note I think if core build makefile could somehow use Compilation Database Parser instead of parsing the output (as it can be configured in Managed Build) then issue with compiler command to language mapping would still exist (since there will be g++ -x c++ again) but I will not need to build the project to do that.

Command line parsing is a bit brittle and all of build systems I frequently encounter already have compile_commands.json generator (these include autotools via bear, CMake and Meson) so ideally there should be a way to utilize compilation database parser in all of build process implementation of CDT as alternative to build output parser. I do not see that for core build makefile, but it exists for managed build makefile.

Now probably back to square one, let me try to reword: I would like CDT to somehow define language of resource based on the compiler used by build system, and doing that preferrably by utilizing compile_commands.json which always exists in my environments.

[ewaterlander]

You are right, we need compile_commands.json based indexing for Core Build Makefile projects too. It is not there at the moment.

Development on the classic indexer has stopped. The focus is now on the new LSP editor, which uses compile_commands.json based indexing with clangd. This works for CMake projects, but for Makefile projects it needs more work. I never tried a Makefile project with compile_commands.json.

Why can't you use the LSP editor?

@ghentschke have you tried LSP editor and Makefile projects?

[i-garrison]

Why can't you use the LSP editor?

Keen to see LSP based one to get on par some time in the future though, and also I'm actively exploring opportunities to improve it. Other than that, well, I tried LSP editor (and also a few of LSP-based commercial ones) but compared to CDT editor with original indexer (once you make it work, but that's a different story) - they all lack quite a lot for me to use them on daily basis.
I'm mostly concerned about whether current call hierarchy feature of original CDT editor will be replicated at all. Here are a couple picks with tickets (I did not looked at current state of those but should be up-to-date):

• variable references in call hierarchy Show variable references in call hierarchy cdt-lsp#272
• function signatures missing in clangd data could call hierarchy view show function signature, such as return or parameter type? clangd/clangd#915
• Extra detail for callHierarchy - really needed, function signatures and stuff Extra detail for callHierarchy clangd/clangd#1940
• I also filed a proposal to extend the protocol with generic array of symbol attributes just in case it will help with symbol details in call hierarchy view Proposal: to better structure symbol details add LSPArray languageSpecificDetails property to callHierarchy and documentSymbol etc. microsoft/language-server-protocol#2118

These are likely just a tip of the iceberg and already require cooperation from clangd team and cdt-lsp team to go through all of the required bits to achieve something comparable to what you can do with current CDT editor: just open sizeable project, navigate to a variable and ctrl+alt+H to focus on what's going on in that area, including all of the callers etc. and most importantly who cares to write the variable, all in single compact view.

Development on the classic indexer has stopped.

I do not think I have seen official notice on that, so until patches are accepted at all I probably will be able to contribute some improvements to current indexer. Fortunately the actually used subset of latest and greatest C++ language is rather compact if you look at real C++ projects, but there is a limit of what we can do with current CDT indexer so again keen to see cdt-lsp progress.

[ghentschke]

@ghentschke have you tried LSP editor and Makefile projects?

Yes, I tested it a litlte and I've noticed that the created compile_commands.json is sometimes faulty. See #1138

I must admit that there is still a feature gap form the old to the new C/C++ editor. Especially for multi projects and/or mixed C/C++ (like gdb) projects as well as for Makefile projects.
CTD-LSP strongly relies on feature implemented in the Language Server Protocol (LSP) and clangd. So the influence on when and which features are implemented is limited.

IMO it could be a good strategy to rely for some features on the CDT indexer until its supported by clangd and/or LSP. The software architecture should be made in a way that it should be made easy to switch the information provider from CDT internal stuff to LSP/clangd based in the future.

[i-garrison]

IMO it could be a good strategy to rely for some features on the CDT indexer until its supported by clangd and/or LSP. The software architecture should be made in a way that it should be made easy to switch the information provider from CDT internal stuff to LSP/clangd based in the future.

Hi @ghentschke do you believe we can actually produce IASTTranslationUnit out of AST obtained via LSP?

I think most of issues with original CDT if used as C++ IDE for modern C++ sources are due to parsing errors. Just looked at AST from clang -x c++ -Xclang -ast-dump -fsyntax-only now and it is very similar in structure (not surprisingly) so it might be not hard to produce IASTTranslationUnit out of it. I would not oppose the idea of keeping the original indexer database to have all of existing features working until replacement is just as good as current CDT.

I can probably join the experiment to have such alternative parser but my java-foo is not good enough to get started. I think for given source file we need some form of AST produced with all required flags and macros, ideally using existing compile_commands.json for that source file, and then run through that and build the IASTTranslationUnit and populate indexer database as needed. If such AST is available I probably can give this transformation a try.

BTW just came across related discussion on cdt-dev mailing list https://www.eclipse.org/lists/cdt-dev/msg35455.html and now I'd like to explore producing CDT AST out of clang AST as stop-gap measure.

[ghentschke]

Hi @ghentschke do you believe we can actually produce IASTTranslationUnit out of AST obtained via LSP?

I think this should be tested. But my time is unfortunately very limited. I am currently focusing on bug fixing and cdt-lsp stabilization.