How to build elementum binaries properly?

[antonsoroko]

Hi.
Is there any tricks to build elementum binaries properly?
I can see old build steps in https://github.com/elgatito/elementum/blob/master/.gitlab-ci.yml
and new one in https://travis-ci.org/github/elgatito/elementum/jobs/471307519/config
and some quick start info in https://github.com/elgatito/elementum/blob/master/README.md and in https://github.com/ElementumOrg/libtorrent-go/blob/master/README.md and in https://github.com/elgatito/elementum/blob/master/Makefile

So I was able to build elementum (and binary works), but I had some small issues, that's why I am curious.
e.g. on go get -u ./... go tried to build libtorrent-go and I got Package libtorrent-rasterbar was not found in the pkg-config search path. which I ignored since it should be build differently as I can see.
But I do not see this issue in travis-ci log for example.

Also I do not see how you build libtorrent-go in travis-ci - no make libs or similar.

I would appreciate some info. Thank you!

p.s. this might be useful for #536

[elgatito]

@antonsoroko I'm not sure travis build is working, that was long time ago.
Basically libtorrent-go's make env is compiling every dependency in a docker image, so when you do make in elementum, it will build in the docker image for specific platform.

There is a make local-env target in libtorrent-go, that will compile everything locally without docker so then you will be able to compile it without docker.

[antonsoroko]

ahhh.
so do i understand correctly that nowadays you prepare releases manually, right?

still interesting how travis-ci still works, maybe it has some old libtorrent-go binaries from cache https://travis-ci.org/github/elgatito/elementum/jobs/471307519#L456

[elgatito]

@antonsoroko

so do i understand correctly that nowadays you prepare releases manually, right?

Almost. Local Jenkins.

Up-to-date libtorrent-go docker images are uploaded to docker hub. When you do "make pull-all" from libtorrent-go repo - it will download docker images so you don't need to build them.

[antonsoroko]

i see. thanks. would be nice to save jenkins job config to github, just in case.

[antonsoroko]

@elgatito
I have an interesting question:
How do you manage to see what real c++ libtorrent calls are used via libtorrent-go?
I mean something like https://github.com/elgatito/elementum/blob/33bb72810ca24ad182ccd97d80f3622629ea6a64/bittorrent/service.go#L647

Since we don't have intermediate generated files, tools like gopls complain about missing functions, e.g. NewTorrentInfo not declared by package libtorrent.

So far i managed to obtain intermediate generated files by editing
https://github.com/ElementumOrg/libtorrent-go/blob/196218fc60e5bd5948c91338edb18e37e4c2d9d8/Makefile#L155
and adding -work to go install so then (after running commands from "Build locally without Docker" section from Readme.) I can copy these 2 files:
_libtorrent_swig.go
libtorrent_wrap.cxx

and use them as a reference code.

Another option is to directly run swig like swig -c++ -DTORRENT_NO_DEPRECATE -DTORRENT_USE_OPENSSL -DBOOST_ASIO_HASH_MAP_BUCKETS=1021 -DBOOST_EXCEPTION_DISABLE -DBOOST_ASIO_ENABLE_CANCELIO -I./local-env/include -I./local-env/include/libtorrent -go -cgo -intgosize 64 libtorrent.swigcxx (i took parameters from make file).

I also tried to copy all files (i mean also files generated by cgo - otherwise gopls generates a lot of errors) from go install -work into something like ~/go/pkg/mod/github.com/!elementum!org/[email protected]/ so then gopls can show me the code, but this breaks compilation.

So currently i just search these 2 files manually.

[elgatito]

@antonsoroko Yes, I have same issue. When you do CGO you don't have intermediate cxx/go files. It just makes a compiled package inside Go pkg directory.

It was somewhen working with proper syntax highlighting and code completion, now it is not working on my dev host.

[antonsoroko]

@elgatito
in order to (dramatically) speed-up local builds i changed https://github.com/elgatito/elementum/blob/9528788f6f9a689cb35c9c0a6b25e2909d182062/Makefile#L173
to use GOCACHE

$(DOCKER) run --rm -v $(GOPATH):/go -e GOPATH=/go -v $(shell pwd):/go/src/$(GO_PKG) -v $(shell go env GOCACHE):/root/.cache/go-build --ulimit memlock=67108864 -w /go/src/$(GO_PKG) $(DOCKER_IMAGE):$(TARGET_OS)-$(TARGET_ARCH) make dist TARGET_OS=$(TARGET_OS) TARGET_ARCH=$(TARGET_ARCH) GIT_VERSION=$(GIT_VERSION)
sudo chown -R $(shell id --user):$(shell id --group) $(shell go env GOCACHE)

but i am not sure that this will not break your CI, so i have not sent patch yet.

i guess for CI the best option would be https://www.docker.com/blog/containerize-your-go-developer-environment-part-2/

but if my approach will work for your CI then i guess we can keep it simple.

[antonsoroko]

@elgatito

looks like in order to use "cache volume" image also should be changed first. not cool.

so, maybe let's add

$(DOCKER) run --rm -v $(GOPATH):/go -e GOPATH=/go -v $(shell pwd):/go/src/$(GO_PKG) -v $(shell go env GOCACHE):/root/.cache/go-build --ulimit memlock=67108864 -w /go/src/$(GO_PKG) $(DOCKER_IMAGE):$(TARGET_OS)-$(TARGET_ARCH) make dist TARGET_OS=$(TARGET_OS) TARGET_ARCH=$(TARGET_ARCH) GIT_VERSION=$(GIT_VERSION)
sudo chown -R $(shell id --user):$(shell id --group) $(shell go env GOCACHE)

to Makefile? if it will not break your Jenkins CI of course,

it is really very convenient.

[elgatito]

@antonsoroko what exactly to add?

[antonsoroko]

diff --git a/Makefile b/Makefile
index f52075c..90f407f 100644
--- a/Makefile
+++ b/Makefile
@@ -170,7 +170,8 @@ build: force
ifeq ($(TARGET_OS), windows)
GOOS=windows $(GO) get -u github.com/StackExchange/wmi
endif
-       $(DOCKER) run --rm -v $(GOPATH):/go -e GOPATH=/go -v $(shell pwd):/go/src/$(GO_PKG) --ulimit memlock=67108864 -w /go/src/$(GO_PKG) $(DOCKER_IMAGE):$(TARGET_OS)-$(TARGET_ARCH) make dist TARGET_OS=$(TARGET_OS) TARGET_ARCH=$(TARGET_ARCH) GIT_VERSION=$(GIT_VERSION)
+       $(DOCKER) run --rm -v $(GOPATH):/go -e GOPATH=/go -v $(shell pwd):/go/src/$(GO_PKG) -v $(shell go env GOCACHE):/root/.cache/go-build --ulimit memlock=67108864 -w /go/src/$(GO_PKG) $(DOCKER_IMAGE):$(TARGET_OS)-$(TARGET_ARCH) make dist TARGET_OS=$(TARGET_OS) TARGET_ARCH=$(TARGET_ARCH) GIT_VERSION=$(GIT_VERSION)
+       sudo chown -R $(shell id --user):$(shell id --group) $(shell go env GOCACHE)

docker: force
$(DOCKER) run --rm -v $(GOPATH):/go -v -e GOPATH=/go -v $(shell pwd):/go/src/$(GO_PKG) --ulimit memlock=67108864 -w /go/src/$(GO_PKG) $(DOCKER_IMAGE):$(TARGET_OS)-$(TARGET_ARCH)

[elgatito]

@antonsoroko try the latest commit in elementum repository. Docker there is ran by your local user, and all the gocache data is also owned by you.

[antonsoroko]

works nice. and now build/ is also not owned by root. thank you!