add support for "garble list"?

[mvdan]

It could be useful for tinygo build, for example, as it uses go list -json -deps to obtain the source code of the input packages. We could perhaps shoehorn garble in there by making garble list provide the equivalent but with obfuscated files and package paths.

I imagine that garble list could also be useful for debugging or inspecting how the obfuscator works without doing full obfuscated builds.

Would it be useful for anything else? This command not be a huge amount of work to write and maintain, but I also want to make sure that people want it before we invest time into it.

[mvdan]

I should note that I'm not sure how to cache the obfuscation process involved in garble list. It's easy for garble build to do that, since we only compile the packages missing from the build cache, so we only need to obfuscate those. But for garble list, I need to show the obfuscated source for all listed packages, even if they're present in the build cache.

I guess that's not a big deal since our obfuscation process isn't expensive when compared to compilation. But still worth noting.

[aykevl]

Reading the above, it appears that garble directly reads the Go build cache? Is that true?

TinyGo does not use the Go build cache, instead it implements its own cache. And this cache should be considered an implementation detail. How TinyGo implements the cache is (in short) by hashing all input files and using that as a cache key, something similar might work for garble.

[mvdan]

When obfuscating+compiling a package, garble does read from the build cache to load type information from dependencies.

However, garble doesn't store anything in the build cache itself. go build simply doesn't re-compile the obfuscated packages which are already built, so garble doesn't need to re-obfuscate them.

So, naively, garble list -deps would have to obfuscate all the source files at every run, unless we implement our own caching, like you say.

How TinyGo implements the cache is (in short) by hashing all input files and using that as a cache key, something similar might work for garble.

By the way, starting in Go 1.16 you could even reuse Go's own cache key:

$ go list -deps -export -f '{{.ImportPath}} {{.BuildID}}' time
unsafe 
internal/unsafeheader KOnHqt6V9qwbBEJxInmw/9eedXXBaCKxwT1ljSQKu
internal/abi Mzptfn-5ExOHbHcpVvwV/LlkOtTU0te3XpZZpgtke
internal/cpu wvbKxFpkMKDKVF9s8Tv1/KgvcD1A_XJn_3tYKcwKT
internal/bytealg -jCcuO5GtgfYh1qFvHq8/5NVDzomoYIPapTf2_YoG
runtime/internal/atomic xQtYUYu95saqTgcYTuyw/JKrSBwqZLVDbqNkHCZ-V
runtime/internal/sys BnzD83fhEi8klbnB4fsu/o9aycGy562yw2grorpIe
runtime/internal/math vEWaq4fX9pm-jXCCqaIH/Z5MKKBvSq4yNQCUuZHW9
runtime 1Z7jsH6ofg8w3Ht8p-e0/CboEGAdK_BZ7vyG3OFyI
internal/reflectlite uIzQltC7yoIA_98hxEtt/50y9n0R5Zdwy-n08HW_R
errors LWeVGES9bXwnxAycUgI7/BHqgcQiTGimEmXcmFdvs
internal/race RwGLGFLse-lujux7pkq2/VAiZiQaUR8cnCaK7HVay
sync/atomic XHhuPKufGrXrj4_-Zp1t/ofBP534cxi9O9-Gq85zV
sync iiqm0lqTwbI-1LL7ONtt/Gf927sWQnu3g98vvr807
internal/itoa 0LRebLIHeg0p_Jat0Qcd/I_qEQnsq4j3agUgkEaDV
internal/oserror jaKBjf3bxG-2cweXXIQ8/f1Q6lBQI8dBpwTPptzmr
syscall -wBss406yXaHGhhRraKx/oaoKePC8JOiexfkPfn_s
time 5K_EPp4evDZd7iz5Gd89/rOEvPm_6Ad6oiXgDfRbp

The hash is split into two parts, where the first part is the hash of the package's inputs (source files, deps, compiler version, etc), and the second hash is the hash of its build result - the compiled object file.

This requires -export because the "deps" above is the hash of the build result of each dep, so it must make sure packages are built to work out the cache IDs. This should be fast once the build cache is warm, but beware that if a dependency changes, then you'd be rebuilding all the dependents to work out the new IDs.

[mvdan]

Also, the reason rebuilds are needed is for reproducibility with bootstrapping; see https://golang.org/src/cmd/go/internal/work/buildid.go.

[mvdan]

For the sake of search engines and indexers, another useful keyword here is gomobile, which also replaces go build.