Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go1.10 and the pkg cache #500

Open
keegancsmith opened this issue Feb 14, 2018 · 17 comments
Open

Go1.10 and the pkg cache #500

keegancsmith opened this issue Feb 14, 2018 · 17 comments

Comments

@keegancsmith
Copy link

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.

From https://tip.golang.org/doc/go1.10#build

If I am not mistaken gocode uses the pkg dir to help complete. One specific example I know of is completing an import. It seems the pkg directory may be populated less often, so will not be as reliable for helping with completion. I am unsure if this will actually affect gocode in practice, just filing to open the discussion.

@nsf
Copy link
Owner

nsf commented Feb 14, 2018

Yes, it sucks. With gocode you have to use "go install" anyway and that package cache feature is useless to you. Sadly there are no plans on my side to workaround it. I talked a bit about it a few times. A proper autocompletion service should use source files instead of package files. Somebody should make one. My enthusiasm for gocode is long gone.

I would suggest looking for alternatives. However I'm unaware of their state. I know there were few attempts of making "language server protocol" servers for Go. Have no idea how good they are. Sadly proper autocompletion tool of a kind should reimplement all language semantic analysis, because working with code that is being actively edited is a slightly different task that doing compilation or static analysis. I think I know how to make a proper service here, but... There's always but.. Maybe one day.

keegancsmith added a commit to sourcegraph/go-langserver that referenced this issue Feb 14, 2018
go1.10 introduces the cache directory. So now objects will only appear in `pkg` if
they are explicitly installed. Previous versions of go would create entries for
the transitive dependencies. So for our vendor tests running `go install -i ...`
would not create a corresponding file in `pkg` for the vendored packages since
`...` does not list vendor. Using `all` instead does include the vendored pkg,
so will include the vendored pkgs. This will likely have wider effects on go
tooling that rely on pkg. Already filed nsf/gocode#500
for gocode.
@keegancsmith
Copy link
Author

I'm actually one of the maintainers of https://github.com/sourcegraph/go-langserver We had a nice contribution which embedded gocode for completion, so we just use gocode in that case. Yes it will require quite a lot of effort to do properly, our language server relies on (mostly) glueing stuff that already exists in go/x/tools and other great OSS projects to work. I believe there is work which hopefully is fruitful for a langserver from scratch at dominikh/go-tools#136

I actually noticed this due to it failing one of the langserver regression tests on go1.10 See sourcegraph/go-langserver@db373e4 for more info.

@nsf
Copy link
Owner

nsf commented Feb 14, 2018

Well ok, I'll have a closer look at this package cache feature (sooner or later). Will let you know then, maybe it's possible to make gocode work with that cached stuff as well.

keegancsmith added a commit to sourcegraph/go-langserver that referenced this issue Feb 14, 2018
* Test tip on travis

* Only test if godef to GOROOT matches path

* Install all instead of ...

go1.10 introduces the cache directory. So now objects will only appear in `pkg` if
they are explicitly installed. Previous versions of go would create entries for
the transitive dependencies. So for our vendor tests running `go install -i ...`
would not create a corresponding file in `pkg` for the vendored packages since
`...` does not list vendor. Using `all` instead does include the vendored pkg,
so will include the vendored pkgs. This will likely have wider effects on go
tooling that rely on pkg. Already filed nsf/gocode#500
for gocode.

* Handle differing diagnostic output

* Correctly remove range in def test on windows
@DisposaBoy
Copy link
Contributor

DisposaBoy commented Feb 17, 2018

FWIW, I've been using go1.10 and gocode with the autobuild option for about a month and haven't noticed any issues.

...but i regularly destroy the pkg directory and recompile things so YMMV

EDIT: also note that go install now has a new option -i so go install -i should restore the old behaviour.

@nsf
Copy link
Owner

nsf commented Mar 24, 2018

I'm not going to do anything about it for now. Yeah, I know it sucks. Sorry about that. I really dislike this idea with package cache. It feels like another attempt to fix their broken package management system.

and the very concept of installed packages may disappear in a future release

Good luck with that, I'm out of here...

No, just kidding. Well, I talked about that in my "gocode rewrite" issue. The problem is that a proper tool should work with source files. Hence, there are two options for me:

  1. Make gocode work with source files.
  2. Rewrite it completely.

Problem: lack of motivation and free time. I definitely favor second option, but it requires way more time than the first one.

I don't know what else to say.

@zchee
Copy link
Contributor

zchee commented Mar 24, 2018

@nsf Hmm...
The time to write gocode v2 came? If you really lack of motivation and free time, As I said before, I want to try to develop the alternative gocode.

What do you think about it? (I mean, the situation of Go core changes)

@nsf
Copy link
Owner

nsf commented Mar 24, 2018

You don't need to ask me. If you can and want to make something that works well - do it. However, I may end up doing gocode v2 anyway one day.

I wrote gocode for myself and when time comes and gocode stops working for me, I'll look for alternatives. In case if I don't find one, I'll make whatever tool is necessary.

I think it's just a question about whether somebody can make something that works well and is willing to maintain it for years without expecting anything in return. So if you feel like doing it, do it. If not - whatever.

I'm not looking for help per se and won't help anyone either.

As for whether it's the time. Maybe, maybe not. I'm still using gb tool mainly (https://getgb.io), hence all that fuss around package management and go tool changes doesn't affect me at all.

@zchee
Copy link
Contributor

zchee commented Mar 24, 2018

@nsf thanks for describing your thought.
Now Go core uses package cache, gocode doesn't support yet. But certainly gocode still fast completion engine for Go. Now didn't time comes and gocode stops working.

You don't need to ask me. If you can and want to make something that works well - do it. However, I may end up doing gocode v2 anyway one day.

Thanks, If I have time, will try to write PoC code. thanks :)

@myitcv
Copy link
Contributor

myitcv commented Mar 28, 2018

@nsf - I think the question I asked in this thread is relevant here.

The one question that comes to mind is how this pipeline of tools will work together and whether the cache resolution step (which ultimately involves hashing of files) adds a significant delay to a gocode answer (because at the moment there is no resolution required). Feels like it won't be insignificant, and that therefore some sort of background ahead-of-time resolution might be required... but honestly I'm guessing without the numbers.

@nsf
Copy link
Owner

nsf commented Mar 28, 2018

I said many times. Tools should work with source files only. The fact that "go/types" depends on compiled packages is a problem that needs to be solved. I don't care about "go/types" that much, don't use it, but other people who write tools might want to push that idea forward. Technically I haven't even looked into "go/types" code, last time I saw it, I remember it had some pluggable package resolvers of some kind. Maybe one can make it work with source files instead of compiled packages.

That's the ultimate solution for tools - just use source files. In particular it's even a requirement for things such as cross-package renaming.

@myitcv
Copy link
Contributor

myitcv commented Mar 28, 2018

As I understand things, go/types does load sources files in the package of interest; it's imports that are loaded via go/types.Importer (which can follow whatever strategy you like, one of which is source file-based: https://godoc.org/golang.org/x/tools/go/loader).

In the fork https://github.com/mdempsky/gocode, @mdempsky made gocode go/types-based (ultimately using the go/loader importer)

Loading everything from source would surely be far too slow in general, no?

@nsf
Copy link
Owner

nsf commented Mar 28, 2018

Using source files might be slower. But not by much.

@myitcv
Copy link
Contributor

myitcv commented Apr 11, 2018

mdempsky#26 is a proof-of-concept of vgo and (by definition since vgo requires Go 1.10) build-cache aware version of gocode.

As the PR mentions, it depends on https://go-review.googlesource.com/c/vgo/+/105855 which is very much WIP itself.

@metakeule
Copy link

Seems like the way to go is to use
golang.org/x/tools/go/packages
see:

golang/go#24661

@thatjames
Copy link

I had this issue but managed to resolve it.

I can't explain what happened, but it appears as if GOPATH/pkg/destdir was not being updated when either go build -i or go install were being called. This is 99% likely due to the change in package caching.

Tanking the GOPATH/pkg/GOARCH/destdir directory and running go build -i in a loop recreated these objects and my autocomplete is working again.

Totally agree with @nsf about the tooling working directly with source code, this is just a workaround for the issue described here. Works well enough for me.

@myitcv
Copy link
Contributor

myitcv commented Jan 14, 2019

https://github.com/stamblerre/gocode is the fork of gocode to use if you need/want Go modules support. This fork is based on go/packages, which uses the go build cache under the covers. This means that you in effect have source code-based completions, because the go build cache is rebuilt as required.

Otherwise https://github.com/mdempsky/gocode is the main fork; this is not based on go/packages and so can suffer from dependencies being out of date.

@thatjames
Copy link

@myitcv Ah, thank you :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants