Tag Lookup Help / Feature Request

[still-dreaming-1]

I'm having a hard time understanding how tags can be useful, and I'm hoping I'm simply unaware of a commonly used way to lookup tags that is better than what I have been doing. if such a tags lookup feature does not exist, this is a suggestion to add such a feature.

Here is my problem with tags. People seem to be really focused on the automated generation a tags file with lots of information to distinguish each tag, rather than an automatically smart way to lookup a tag. I will now rephrase this as an example. It might sound like I'm asking for Neovim help and not tags help, but I will try to explain why I think it should all be handled by the same program, the one capable of generating the tags.

I am primarily interested in tags for the purpose of doing code lookup from an editor, or providing autocomplete information to an editor. In my case I happen to be using Neovim. So let's say I have some file open, and the cursor pointed at a method call. I do a code lookup using tags. At this point, the editor does not take into account the context from which I'm looking up the tag. But why should the editor know how to parse the file and understand the code, when that is what the tags program was good at? If the editor was capable of doing that, it wouldn't need a tag generating program.

From my perspective, the program that is capable of understanding the code well enough to parse it and make tags, needs to be activated when doing a tags lookup, otherwise the lookup will not be very intelligent, or will require the person doing the lookup to write a very specific lookup query based on mentally parsing the code rather than the computer parsing it and writing that tag lookup query for you. So I'm wondering if universal ctags provides that type of lookup feature. One that accepts as its' input the file path and current contents (which may be different from what is saved), and the line and character range, or something like that. It would then activate the same parser it used to generate the tags, and figure out which tag is most appropriate for that context.

Even though that is the type of feature I am looking for, even if it existed, it would still leave me confused about the purpose of tags. Why fire up the parser and parse everything, just to find out how to search the tags file, which was previously generated by firing up the parser and parsing everything? Why not just do a parse and return the information directly rather than from a tags file? I guess there might be efficiency gains by doing a full parse every time the tags file is generated, and then only doing a partial parse to decide how to search the tags file?

[masatake]

Do you know "language server" (https://github.com/Microsoft/language-server-protocol) ?
I guess it is what you want.

To realize what you want, we have to think and redesign the interface between editor and ctags.
Currently, tags file is the interface; ctags provides it and an editor reads it. ctags doesn't consider real-time user interaction. I cannot imagine how can I extend ctags to support real-time user interaction. However,
it is obvious to me that, such extended ctags will not be ctags. It will be completely different software.

Some parser of Universal-ctags provides one critical information about context.

int
main(void)
{

}

For the input ctags can generate:

[jet@localhost]~/var/strace% u-ctags --fields=+en -o - /tmp/foo.c 
main	/tmp/foo.c	/^main(void)$/;"	f	line:2	typeref:typename:int	end:5

end: field is newly added in Universal-ctags. With the information, a client tool like editor can know which definition the current line is. As far as I know, there is no editor using this end: information.

[still-dreaming-1]

@masatake That is kind my point though. It seems the very concept of what ctags is, is fundamentally flawed. How is an editor, a tags generator, and a tags file a helpful combination of things? How can a tags file actually be useful to an editor? It is certain not helpful for doing intelligent code definition lookup or intelligent autocompletion.

Language servers are too complex, slow, and buggy. They have to keep running and they save cache information, which is just begging for trouble. All that is fine if it is designed as part of an IDE where all the tools are made by the same team, and they are all designed to work together, but when you try to get a bunch of random tools to work together like this, well I just have never gotten it to work well for something as complex as intelligent code definition lookup or intelligent autocompletion.

Universal ctags is much more simple in that you just run it and wait for the result to be returned/produced after it is finished. It can also generate tags for my code in an instant. Let's say someone wanted to do a code definition lookup. They should be able to tell ctags to generate a code definition lookup tags file. They would pass to ctags the path to the file they are editing, the current contents of the file (which may be different than what is saved, if they haven't saved the latest changes), and a single character representing where the cursor/mouse is at. That character would clue ctags into what they are looking at that they want to find the definition of. Ctags would then use that exact context to generate a very small, one time use file, or potentially just the data itself, that only contains potential candidates for where the thing they are looking up is defined. Ideally it would be so small that it only contains one candidate. That would be the ideal scenario because it means it successfully figured out exactly what they were actually trying to lookup. If it can't narrow it down to a single absolute match, it should attempt to order the results by an estimated likelihood that this is what they are looking for.

Something similar could be done for an autocompletion tags file. Once again, the contextual understanding would need to happen inside of ctags itself, it doesn't make sense for the editor to do that. It can't be that ctags is asked to provide all the methods for a class, it just needs to be asked to provide an autocompletion file, and ctags would be the tool that determines what they are trying to autocomplete. Once again, the user would pass to ctags a path, the current file contents, and the position of the cursor. It would use that information to produce a one time, throw away, autocompletion tags file, or perhaps just the file contents.

The workflow would be different for the editors because they would not be generating a tags file at certain points and then reusing it. They would ask ctags to generate a new one every time they needed to find the definition of something or get autocompletion information.

This would require the logic of ctags to be changed from generating as much information as possible to trying to figure out what is the only information you are looking for based on a context and a goal.

I don't want ctags to become a language server. I don't want it to run in the background and keep cached information. I want it to continue to be an alternative to language servers. I want it to be run to get some results, and then it should stop running until it is ran again to get some different results. It should not reuse old information from the last time it was ran. There are times this will be too slow for certain users or certain code bases, but this design would still help distinguish it from language servers.

[masatake]

An example is needed.

Using the current ^ as the marker that represents the cursor position of an editor.
Think about python as the target language.
Whole source files in a project are already tagged. The results are stored to a tags file.

# funcX is defined in foo
import foo

# funcX is defined in bar, too
import bar

foo.funcX^()

An unintelligent editor may try to search just funcX in the tags file and find two entries: one in foo and the other in bar.

However, your idea is that an ideall tool, let's call it ztags, can do in following steps:

1. The editor passes ztags the positional information of ^ and the name of target source file.
2. ztags makes a query for searching funcX which is defined in a namespace foo.
3. ztags runs the query.
4. ztags returns the result to the editor. The result just for funcX in foo.

Do I understand your intent correctly?
If I'm wrong, please, show me an example with github's markdown notation.

[still-dreaming-1]

That seems exactly right... I guess I'm just hoping the simplicity of doing this with something like ctags would be more reliable than trying to interface with a language server.