- Introduction
- Guile, the language
- Guix, the language
- Starting the daemon
- Install Guix from the git repository
- Creating a package
- Building the package
- Debugging the package
- Fixing problems
- Installing the package
- Using the search path
- Making a patch and submit to the Guix project
- Workflow for packaging
- Hacking tips
- Hints
- The Ruby package
- Dealing with special packages
- Create a caching server
‘You are in a maze of twisty packages all alike…’
Hacking GNU Guix is an adventure. Not least because it is using Scheme LISP in the GNU Ubiquitous Intelligent Language for Extensions implementation, also known as Guile. You are encouraged to dive in to LISP, a language that is so good it simply refuses to go away.
GNU Guix stands out as the ‘hackable’ package manager. Mostly because it uses a powerful high-level programming language.
This document should get you started with Guile and GNU Guix. Just pick any package description in the ./gnu directory and work from there. The examples here are pulled from the ruby.scm package.
Once you have a running GNU Guix (see INSTALL), the next step is to compile a package from a recent git checkout of the sources. Check out the source tree following the instructions in the manual. Here we start from a checked out GNU Guix git repository.
First a note of caution. Try to work one step at a time. GNU Guix is not a simple system, so you are bound to get frustrated. But it is hackable, that means you can solve any problem! And the reward is sane software deployment. Guix will pay back your investment.
Before hacking GNU Guix it may be worth contemplating the speed of the network connection: in short, the faster the better. Caching packages somewhere may be worth considering too. Finally a fast server may be a good thing too because GNU Guix is designed to build packages in parallel.
Guile is a Scheme LISP. Here we list some Scheme specific material that is used by GNU Guix. There is much information on the web on Scheme. Check out Scheme at a Glance, for example.
In the Scheme world, we prefer to use the word `procedure’ instead of `function’, to reflect the fact that it is not merely a mapping from inputs to outputs but can perform side effects as well.
In this document I carelessly mix the two terms, you are warned.
#f signifies false, #t signifies true.
The #: signifies literal keyword syntax and is used to create unique identifiers, see also
http://practical-scheme.net/wiliki/schemexref.cgi?keyword%3F
in Ruby, for comparison, #:key would be in colon notation :key (which is known as a symbol in Ruby, but differs from a symbol in LISP).
The percentage is a syntactical name helper used to generate and create values available in scope. For example, the output file name is reified via the %output variable automatically added to builder’s scope. Input file names are similarly reified through the %build-inputs variable. Both variables are non-hygienically introduced in the build expression by build-expression->derivation.
Note that it is merely a convention, like ‘_’ in C. Scheme LISP treats ‘%’ exactly the same as any other letter.
GNU Guix uses key-value pairs extensively. With
(build-system
(name 'ruby)
(description "The standard Ruby build system")
(lower lower))the Guix build-system record constructor is called with the field names name, description and lower, where the last is a function with the same name. These definitions are actually resolved as Guile records can be found in ./guix/packages.scm. Look up ‘define-record-type*’ defined in ./guix/build-system.scm to see how that works (the asterisk * implies that fields are bound as per letrec*, allowing them to refer to one another):
(define-record-type* <build-system> build-system make-build-system
build-system?
(name build-system-name) ; symbol
(description build-system-description) ; short description
(lower build-system-lower)) ; args ... -> bags
;; "Bags" are low-level representations of "packages". The system and target
;; of a bag is fixed when it's created. This is because build systems may
;; choose inputs as a function of the system and target.
(define-record-type* <bag> bag %make-bag
bag?
(name bag-name) ;string
(system bag-system) ;string
(target bag-target ;string | #f
(default #f))
;; Here we use build/host/target in the sense of the GNU tool chain (info
;; "(autoconf) Specifying Target Triplets").
(build-inputs bag-build-inputs ;list of packages
(default '()))
(host-inputs bag-host-inputs ;list of packages
(default '()))
;; "Target inputs" are packages that are built natively, but that are used
;; by target programs in a cross-compilation environment. Thus, they act
;; like 'inputs' as far as search paths are concerned. The only example of
;; that is the cross-libc: it is an input of 'cross-gcc', thus built
;; natively; yet, we want it to be considered as a target input for the
;; purposes of $CPATH, $LIBRARY_PATH, etc.
(target-inputs bag-target-inputs
(default '()))
(outputs bag-outputs ;list of strings
(default '("out")))
(arguments bag-arguments ;list
(default '()))
(build bag-build)) ;bag -> derivationIn GNU Guix the record data is available as build-system-name, build-system-description etc. Same for the package record which delivers package-name, package-version, etc.
Also literal keyword syntax is used, e.g.,
(build-expression->derivation store name builder
#:inputs inputs
#:system system
#:modules imported-modules
#:outputs outputs
#:guile-for-build guile-for-build))calls build-expression->derivation (note that Guile can use more than alphanum characters to create a function name) with parameters store, name, builder and a list of variable key-value pairs named #:inputs, inputs etc. The idea is that the number of parameters is variable to the build-expression->derivation function.
define and define* are used to define functions - well actually to bind identifiers to any value. Note that functions are defined in a module or function local scope. define-module at the top of a package can export functions, e.g.
(define-module (guix build-system ruby)
#:use-module (guix store)
#:export (ruby-build
ruby-build-system))The difference between define and define* is that the latter can handle variable length parameter lists.
A thing to note is that every LISP function returns a value, i.e., the last expression evaluated.
let and let* allow defining multiple variables in scope. The difference between let and let* is that let* guarantees sequential initialization, so you can cross-reference values in the list. The more important difference between let and let* is that let* allows the initializers of later variables to refer to the earlier variables, whereas the initializers of let only see variables outside of the let. For example:
(let ((a 1) (b 2))
(let ((b a) (a b))
(list a b)))
returns (2 1), but if the inner let is replaced with let*, then it
will return (1 1).One thing to note is the extensive use of backquote in GNU Guix. Backquote (quasiquote in Scheme LISP jargon) is like quote, but selected subexpressions are evaluated. These are assigned with a comma (an unquote), e.g.
(ruby-build #:name ,name
#:source ,(match (assoc-ref inputs "source")
(((? derivation? source))
(derivation->output-path source))
((source)
source)
(source
source))
#:system ,system
#:test-target ,test-target
#:tests? ,tests?
#:phases ,phases)Note match operator which is used for expression matching. Here ‘source’ is matched to pull out the source path and generate a #:source key-value pair.
When ,@ is used (shorthand for unquote-splicing), e.g. in
(host-inputs `(,@(if source
`(("source" ,source))
'())
,@inputs
;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))it indicates an expression to be evaluated and the elements of the returned list inserted (the resulting list is ‘spliced in’).
Use your editor to jump to function definitions inside the GNU Guix source tree. With emacs you can use ‘ctags -R -e’ in the base directory and load the TAGS file. Jump to a tag with M-x find-tag. If that does not find the tag, look the function up in the Guile manual.
Guile/scheme is a minimalistic implementation of LISP (though Guile is moderately large for a Scheme). This means it is pretty easy to learn the language. To read up on available functionality, read the Guile manual online or in PDF. The procedure index contains all available function calls for the language.
Running Guile stand-alone is easy using a command line REPL or inside emacs. That allows you to play with language features, as well as call GNU Guix functionality directly.
Emacs has a range of packages to deal with Lisp syntax. My favorites are:
- Rainbow delimiters (color brackets)
- Guile mode
- Paredit (auto bracket completion)
- Geiser (REPL)
More tips here.
GNU Guix is not a language per se. But as they say, LISP is used to create a new language for every purpose (using macros). So here we list some of the commonly used macros.
A good explanation of expressions (a derivation in Nix-speak) and how they are implemented can be found on Wikisource. Actually at the low level an expression returns a derivation variable or structure.
Recently GNU Guix introduced bags as an intermediate form between packages and derivations. A bag includes all the implicit inputs which is useful for processing.
Replace the install phase with a function that adds /bin to outputs and makes sure to make the directory and copy a file named mpc123 into bin:
;...
(build-system gnu-build-system)
(arguments
'(#:phases
(modify-phases %standard-phases
(delete 'check) ;; Don't run the 'make check' step of the gnu-build-system
(replace 'install ;; Replace the install step with the function defined below
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))
(bin (string-append out "/bin")))
(mkdir-p bin)
(copy-file "mpc123" (string-append bin "/mpc123"))))))))
;...Do not forget to start the daemon
guix-daemon --build-users-group=guix-buildThe daemon runs ar root, the actual build processes run as unprivileged users.
See the section Building GNU Guix from source in INSTALL.
The version is located in the package definition. E.g.
(define-public ruby-2.1
(package (inherit ruby)
(version "2.1.6")
(source
(origin
(method url-fetch)
(uri (string-append "http://cache.ruby-lang.org/pub/ruby/"
(version-major+minor version)
"/ruby-" version ".tar.bz2"))
(sha256
(base32
"1r4bs8lfwsypbcf8j2lpv3by40729vp5mh697njizj97fjp644qy"))))))guix download http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.3.tar.gzif you have downloaded a package or checked out a git repo you can also do
~/.config/guix/current/bin/guix hash /gnu/store/ddg95a3q30qiiqz4gdkmmldj46s9bfmp-gemma-gn2-0.98-6b1e007-checkout -rGuix can read package definitions from other sources and write a Guix expression to stdout. Make sure gnutls is installed (to avoid a JSON error) and
guix package -i gnutls guix import pypi readline
prints out
(package
(name "python-readline")
(version "6.2.4.1")
(source
(origin
(method url-fetch)
(uri (string-append
"https://pypi.python.org/packages/source/r/readline/readline-"
version
".tar.gz"))
(sha256
(base32
"01yi9cls19nglj0h172hhlf64chb0xj5rv1ca38yflpy7ph8c3z0"))))
(build-system python-build-system)
(inputs
`(("python-setuptools" ,python-setuptools)))
(home-page
"http://github.com/ludwigschwardt/python-readline")
(synopsis
"The standard Python readline extension statically linked against the GNU readline library.")
(description
"The standard Python readline extension statically linked against the GNU readline library.")
(license #f))All software (except for the Linux kernel) depends on other software to build or to run. Guix keeps track of them and by adding a dependency all underlying dependencies get pulled in too. The build systems will pull in the usual dependencies, but often you need to specify a few more. Guix understands the following inputs
- native-inputs: required for building but not runtime - installing a package through a substitute won’t install these inputs
- inputs: installed in the store but not in the profile, as well as being present at build time
- propagated-inputs: installed in the store and in the profile, as well as being present at build time
From a prebuilt guix in the source tree one can start with
./pre-inst-env guix package -A ruby
ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2
ruby 2.1.6 out gnu/packages/ruby.scm:91:2
ruby 2.2.2 out gnu/packages/ruby.scm:39:2to see if the package compiles. Note that Guix contains three versions of Ruby! Next try the explicit package compile which should return the destination
./pre-inst-env guix build -K -e '(@ (gnu packages ruby) ruby-2.1)'
/gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6You can find the log files generated during the build process with
guix build --log-file something
Ricardo wrote: does this also work for failed builds - without rebuilding it again? It does seem to work. To test this I added (error “foo”) to a build phase in the “diamond” package and ran
guix package -i diamond
This ends with
Build failed: /gnu/store/wk9qbhmdzs62mp40casrndcgm3p50m3b-diamond-0.9.22.drv guix package: error: build failed: build of `/gnu/store/wk9qbhmdzs62mp40casrndcgm3p50m3b-diamond-0.9.22.drv' failed
So I ran
guix build --log-file /gnu/store/wk9qbhmdzs62mp40casrndcgm3p50m3b-diamond-0.9.22.drv
which gave me
/var/log/guix/drvs/wk/9qbhmdzs62mp40casrndcgm3p50m3b-diamond-0.9.22.drv.bz2
which contains the build log for this failed build, including the “foo” error message.
I would like this error log file location to be shown unprompted, but I think we would need to change build.cc, so that BuildError prints it in addition to the error message.
Before debugging it is important to have a clean environment.
You can view the environment variable definitions Guix recommends with
guix package --search-paths
Mine looks like:
set|grep guix
ACLOCAL_PATH=/home/pjotr/.guix-profile/share/aclocal
BASH=/home/pjotr/.guix-profile/bin/bash
CPATH=/home/pjotr/.guix-profile/include
GUILE_LOAD_COMPILED_PATH=/home/pjotr/.guix-profile/share/guile/site/2.0
GUILE_LOAD_PATH=/home/pjotr/.guix-profile/share/guile/site/2.0
LIBRARY_PATH=/home/pjotr/.guix-profile/lib
LOCPATH=/home/pjotr/.guix-profile/lib/locale
PATH=/home/pjotr/.guix-profile/bin:/home/pjotr/.guix-profile/sbin
PKG_CONFIG_PATH=/home/pjotr/.guix-profile/lib/pkgconfigWith most packaging systems the only way to debug them is by sprinkling print statements, using a debugger or hoping for the best (TM). The equivalent in a guix expression would be, for example
(pk 'ECHO (which "echo"))GNU Guix is written in scheme lisp with the GNU Guile interpreter/compiler. This means code can be run and data can be inspected in the REPL.
From the command line with guile use the REPL like this:
$ ./pre-inst-env guile
GNU Guile 2.0.11
Copyright (C) 1995-2014 Free Software Foundation, Inc.
Enter `,help' for help.
scheme@(guile-user)>
;;; read-line support
(use-modules (ice-9 readline))
(activate-readline)
;;; help may come in useful
,help
;;; some LISP
(define a 3)
a
;;; $1 = 3
,pretty-print a
;;; $2 = 3Load guix (the leading comma interprets the command)
,use (gnu packages ruby)
,use (guix)
,use (guix build-system)Note that the order of gnu/packages/ruby is simply the directory structure of the git repository. Now start talking with the daemon
(define s (open-connection))
ruby
;;; $1 = #<package ruby-2.2.2 gnu/packages/ruby.scm:39 2ed9f00>
ruby-2.1
;;; $1 = #<package ruby-2.1.6 gnu/packages/ruby.scm:91 36f10c0>
(package-derivation s ruby)
;;; $2 = #<derivation /gnu/store/cvsq4yijavhv7vj7pk3ns0qmvvxdp935-ruby-2.2.2.drv => /gnu/store/66nc9miql9frizn0v02iq1siywsq65w5-ruby-2.2.2 3a9d7d0>
,pretty-print s
;;; $3 = #<build-daemon 256.14 32b7800>Let’s inspect the package using the methods defined in guix/packages.scm
(define p ruby)
(package-name p)
;;; "ruby"
(package-inputs p)
;;; (("readline" #<package readline-6.3 gnu/packages/readline.scm:39 2aa2840>)
;;; ("openssl" #<package openssl-1.0.2b gnu/packages/openssl.scm:30 2f15d80>)
;;; ("libffi" #<package libffi-3.1 gnu/packages/libffi.scm:34 2b8b900>)
;;; etc.
(package->bag p)
$22 = #<<bag> name: "ruby-2.2.2" system: "x86_64-linux" target: #f
build-inputs: (
("source" #<origin "http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.2.tar.xz" 6az3luekwvyihzemdwa3zvzztftvpdbxbnte3kiockrsrekcirra () 36f28c0>)
("tar" #<package tar-1.28 gnu/packages/bootstrap.scm:145 3953540>)
("gzip" #<package gzip-1.6 gnu/packages/bootstrap.scm:145 39533c0>)
("bzip2" #<package bzip2-1.0.6 gnu/packages/bootstrap.scm:145 3953240>)
("xz" #<package xz-5.0.4 gnu/packages/bootstrap.scm:145 39530c0>)
("file" #<package file-5.22 gnu/packages/bootstrap.scm:145 395cf00>)
("diffutils" #<package diffutils-3.3 gnu/packages/bootstrap.scm:145 395cd80>)
("patch" #<package patch-2.7.5 gnu/packages/bootstrap.scm:145 395cc00>)
("sed" #<package sed-4.2.2 gnu/packages/bootstrap.scm:145 395ca80>)
("findutils" #<package findutils-4.4.2 gnu/packages/bootstrap.scm:145 395c900>)
("gawk" #<package gawk-4.1.1 gnu/packages/bootstrap.scm:145 395c780>)
("grep" #<package grep-2.21 gnu/packages/bootstrap.scm:145 39536c0>)
("coreutils" #<package coreutils-8.23 gnu/packages/bootstrap.scm:145 3953840>)
("make" #<package make-4.1 gnu/packages/bootstrap.scm:145 3953a80>)
("bash" #<package bash-4.3.33 gnu/packages/bootstrap.scm:145 3953e40>)
("ld-wrapper" #<package ld-wrapper-0 gnu/packages/commencement.scm:644 39539c0>)
("binutils" #<package binutils-2.25 gnu/packages/bootstrap.scm:145 394d3c0>)
("gcc" #<package gcc-4.8.4 gnu/packages/commencement.scm:530 394d180>)
("libc" #<package glibc-2.21 gnu/packages/commencement.scm:454 394d600>)
("locales" #<package glibc-utf8-locales-2.21 gnu/packages/commencement.scm:621 3953c00>)
)
host-inputs: (
("readline" #<package readline-6.3 gnu/packages/readline.scm:39 2aa2840>)
("openssl" #<package openssl-1.0.2b gnu/packages/openssl.scm:30 2f15d80>)
("libffi" #<package libffi-3.1 gnu/packages/libffi.scm:34 2b8b900>)
("gdbm" #<package gdbm-1.11 gnu/packages/gdbm.scm:26 2b8b6c0>)
("zlib" #<package zlib-1.2.7 gnu/packages/compression.scm:33 36f1c00>)
)
target-inputs: ()
outputs: ("out")
arguments: (#:system "x86_64-linux" #:test-target "test" #:parallel-tests? #f #:phases
(alist-cons-before (quote configure) (quote replace-bin-sh)
(lambda _ (substitute* (quote ("Makefile.in" "ext/pty/pty.c" "io.c"
"lib/mkmf.rb" "process.c" "test/rubygems/test_gem_ext_configure_builder.rb"
"test/rdoc/test_rdoc_parser.rb" "test/ruby/test_rubyoptions.rb"
"test/ruby/test_process.rb" "test/ruby/test_system.rb"
"tool/rbinstall.rb"))
(("/bin/sh") (which "sh")))) %standard-phases)
)
build: #<procedure gnu-build (store name input-drvs #:key guile
outputs search-paths configure-flags make-flags out-of-source? tests?
test-target parallel-build? parallel-tests? patch-shebangs?
strip-binaries? strip-flags strip-directories validate-runpath? phases
locale system imported-modules modules substitutable?
allowed-references)>>where bag is the actual data that gets passed to the build system.
Guix uses monad to handle the store state. Read up on these and G-expressions if you intend to hack Guix. To run a procedure within a Store do something like
,use (guix git-download)
(git-reference (url "https://github.com/pjotrp/genenetwork2.git") (commit "860bdcebde5cbb1898c26da80ac67207480c0803"))
$3 = #<<git-reference> url: "https://github.com/pjotrp/genenetwork2.git" commit: "860bdcebde5cbb1898c26da80ac67207480c0803" recursive?: #f>
,enter-store-monad
(git-fetch $3 'sha256
(base32
"0yvkv7pnigvcifas3vcr8sk87xrrb8y9nh9v1yx2p43k0xz1q8vz"))
$4 = #<derivation /gnu/store/fmpk2sck6ny5dgyx12s539qcadzky24n-mypackage.drv => /gnu/store/k6q69arfmsm116a8hfkqqah
m0ddzacjc-mypackage 50b9e10>Here $3 is the git-reference record and $4 is a derivation object, and calling ‘built-derivations’ starts the build process
(built-derivations (list $4))
building path(s) `/gnu/store/fid19bds4rak2zn8pzfhrjdcpmqwhjn4-module-import'
building path(s) `/gnu/store/vf1pmac8yz2g0d4ln5ibwg0xaffdrnpq-module-import-compiled'
building path(s) `/gnu/store/k6q69arfmsm116a8hfkqqahm0ddzacjc-mypackage'
(...)(run-with-store s
(git-fetch ref ...))The principle of a monad is simply to handle `state’ (here the store) outside the called procedures (here the package builder). This prevents passing around state parameters all the time leading to simpler code. For a description of how monads can be implemented in Guile, read Chris Okasaki brilliant writeup `Monadic Programming in Scheme’. If you are a Ruby guy (like me) and want to understand monads, read Tom Stuart’s more gentle `Refactoring Ruby with Monads’.
It is also possible to step through code and view progress and the contents of variables at every stage. The debugger comes with Guile by default. You can set breakpoints and step through code with step, next and finish.
You can set up an init file that gets loaded every time Guile gets started in interactive mode. Mine contains:
;; Init file in ~/.guile
;;; read-line support
(use-modules (ice-9 readline))
(activate-readline)
;;; GNU Guix
(use-modules (guix hash) (guix) (guix build-system))
Instead of using the Guile REPL is is also possible to run the code as a script. Create a script:
(define-module (gnu packages mytest)
#:use-module (gnu packages ruby)
#:use-module (guix)
)
(define s (open-connection))
(define p ruby-2.1)
(write (package->bag p))
(newline)(newline)
(write (string-append (package-name p) "-" (package-version p)))Run it as
./pre-inst-env guile -s test.scm (lots of info)
:
"ruby-2.1.6"
But the best thing, if you use Emacs, is to use Geiser, as noted in ‘HACKING’. In addition to a REPL, it brings stuff like autodoc, jump-to-definition, expression evaluation from the buffer, etc.
Install Geiser and add the guile path to ~/.emacs with
(setq-default geiser-guile-load-path '("~/src/guix"))Start geiser and you should be able to replicate above commands.
Compiling the package there may be build problems. cd into the build directory
cd /gnu/tmp/guix-build-ldc-0.17.2.drv-0and
. environment-variables
makewill recreate the build environment. Now you can see where the build stopped by running commands.
Here I show how you can drill down on tests, disable/fix them fast and create the patch by using ‘git diff’. While this is about the D compiler build system with CMake, the strategy is generic. According to Guix build
The following tests FAILED:
239 - std.datetime (Failed)
299 - std.regex.internal.tests (Failed)
569 - std.datetime-debug (Failed)
629 - std.regex.internal.tests-debug (Failed)
670 - dmd-testsuite-debug (Failed)
673 - llvm-ir-testsuite (Failed)
Using guix build with -K option; I changed into the printed dir after build failure and checked the logs
grep datetime -r *|grep 239
Testing/Temporary/LastTestsFailed.log:239:std.datetime
Testing/Temporary/LastTest.log:239/673 Testing: std.datetime
Testing/Temporary/LastTest.log:239/673 Test: std.datetimeLooking in the log
239/673 Testing: std.datetime 239/673 Test: std.datetime Command: "/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos2-test-runner" "std.datetime" Directory: /gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime "std.datetime" start time: Dec 11 16:16 Europe Output: ---------------------------------------------------------- FAIL release64 std.datetime core.time.TimeException@/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos/std/datetime.d(560): Fractional seconds must be less than one second. ---------------- <end of output> Test time = 0.19 sec ---------------------------------------------------------- Test Failed. "std.datetime" end time: Dec 11 16:16 Europe "std.datetime" time elapsed: 00:00:00 ----------------------------------------------------------
It complains
core.time.TimeException@/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos/std/datetime.d(560):
Fractional seconds must be less than one second.On line 560 we find
enforce(fracSecs < seconds(1), new DateTimeException("Fractional second
s must be less than one second."));First fix of choice: let’s disable this test by commenting it out. But first fix the build dir permissions and start using git
git init
git add runtime/phobos/std/datetime.d
git commit -a -m 'datetime.d'comment out the test and ‘git diff’ should show
- enforce(fracSecs < seconds(1), new DateTimeException("Fractional seconds must be less than one second."));
+ // enforce(fracSecs < seconds(1), new DateTimeException("Fractional seconds must be less than one second."));Next, rerun the test. If you check the Testlog again you can see it can be invoked as
monza:/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime$ ./phobos2-test-runner-debug std.datetimeFirst run make again and rerun the test
make
runtime/phobos2-test-runner-debug std.datetime
****** FAIL release64 std.datetime
core.time.TimeException@/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos/std/datetime.d(560): Fractional seconds must be less than one second.Still complaining! This is we because we also need to build phobos with unittests - unfortunately D creates one huge BLOB of a binary. After some digging in the ctest manual and trial and error I found you can do that by first building the build ‘test’ (as listed by ctest -N):
ctest -R build-phobos2-test-runner-debugupdates runtime/phobos2-test-runner-debug, so now we can
make
runtime/phobos2-test-runner-debug std.datetimeYou may use the additional –build-noclean switch, provided it is the same build you are using (e.g., with or without debug). So, next round
make
ctest -R build-phobos2-test-runner-debug --build-noclean
runtime/phobos2-test-runner-debug std.datetimeshould be faster. But now we got a different error:
****** FAIL release64 std.datetime
core.exception.AssertError@/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos/std/datetime.d(594): assertThrown failed: No TimeException was thrown.
----------------which tests the test we disabled. So we disable that too. And we have success:
monza:/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2$ make
[ 0%] Built target idgen
[ 1%] Built target impcnvgen
[ 16%] Built target LDCShared
[ 16%] Built target ldc2
[ 16%] Built target FileCheck
[ 16%] Built target gen_gccbuiltins
[ 16%] Built target not
[ 18%] Built target ldmd2
[ 18%] Generating std/datetime.o
[ 18%] Linking C static library ../lib/libphobos2-ldc.a
[ 35%] Built target phobos2-ldc
[ 59%] Built target druntime-ldc-debug
[ 59%] Generating std/datetime-debug.o
[ 59%] Linking C static library ../lib/libphobos2-ldc-debug.a
[ 75%] Built target phobos2-ldc-debug
[100%] Built target druntime-ldc
monza:/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2$ ctest -R build-phobos2-test-runner-debug --build-noclean
Test project /gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2
Start 8: build-phobos2-test-runner-debug
1/1 Test #8: build-phobos2-test-runner-debug ... Passed 17.73 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 17.84 sec
monza:/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2$ runtime/phobos2-test-runner-debug std.datetime
****** FAIL release64 std.datetime
core.exception.AssertError@/gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos/std/datetime.d(594): assertThrown failed: No TimeException was thrown.
----------------
See below section on gdb if you get an exception.
It may be some build stuff gets messed up. You can regenerate all relevant binaries with
make clean
make
ctest -R build-phobos2-test-runner-debug\|build-phobos2-ldc-unittest-debug\|build-druntime-test-runner-debug\|build-druntime-ldc-unittest-debug\|std.datetime-debugWhen all tests are ‘fixed’ we can create the patch with
git diff > ldc_disable_failing_tests.patchWhen we have done these we can look at fixing some tests - and perhaps communicating with upstream to see if they want to fix/patch some of these in turn, so we don’t need to redo this work next time round. But at least we can run most of the ldc tests now in Guix.
Note also, because we are using git, we can roll back to an earlier edition of the build dir, e.g., to roll back on changes you have not commited
git reset --hard
In above section I had a segfault at some point and needed to find out where it went wrong. Similar to the earlier command run with gdb in the build directory
~/.guix-profile/bin/gdb --args runtime/phobos2-test-runner-debug std.datetimeAnd inside GDB:
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
(gdb) r
Starting program: /gnu/tmp/guix-build-ldc-0.17.2.drv-0/ldc-0.17.2/runtime/phobos2-test-runner-debug std.datetime
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x000000000082cf9e in std.datetime.SysTime.this(const(std.datetime.DateTime), const(core.time.Duration), immutable(std.datetime.TimeZone)) (
dateTime=<incomplete type>, fracSecs=<incomplete type>,
tz=0x614a2d313030302d, this=...) at datetime.d:567
567 immutable standardTime = nonNullTZ.tzToUTC(adjustedTime.total!"hnsecs");
(gdb)You can spot the problem is at line number 567.
Sometimes you need to modify a source package to compile it on Guix. Here I show my way of creating a patch. This patch with the error log you may want to send upstream to the authors/maintainers, otherwise it will need fixing with every update/release.
First step is to build the package as is with Guix and capture the output so it can be shared. Building from the source tree
./pre-inst-env guix package -i elixir --no-grafts -K &> error.out
You may remove the boiler plate in that file.
To make sure no patches were applied modify the package so the patch does not get applied. In this case comment out
(patches (search-patches "elixir-disable-failing-tests.patch"))))
And rerun the Error log to get a full list of errors.
You may also want to force the build to stop right after unpacking by injecting
(arguments
`(
#:phases
(modify-phases %standard-phases
(add-before 'build 'break (lambda () (#f)))
...Now the build will fail with
ERROR: In procedure #f: note: keeping build directory `/tmp/guix-build-elixir-1.5.1.drv-5'
Another option is to inject (#f)
(replace 'build
(lambda _
(invoke "bash" "-c" "set")
(#f)
(invoke "./build.sh")))will halt after showing the environment
Initialize git using a first terminal
cd /tmp/guix-build-elixir-1.5.1.drv-5 cd elixir* git init
Add the files you are modifying
git add ./lib/elixir/test/elixir/kernel/dialyzer_test.exs git add ./lib/elixir/test/elixir/kernel/cli_test.exs git add ./lib/elixir/test/elixir/system_test.exs git commit -a -m start
Optionally apply the previous patches by hand - we do this now so the become visible in the new patch.
patch -p1 < /tmp/elixir-disable-failing-tests.patch
Hopefully it mostly takes it. Now fix the problems that occur in the source tree and create a new patch using git
git diff > /tmp/elixir-disable-failing-tests-5.patch
Now plug this patch into the source tree again, enable patch application, and retry above steps.
Note: patching can be done incrementally and patches can be merged into one file (by hand). When you get better at this you can probably save on a few build cycles.
Note: always send the errors and patch(es) upstream. Even if they do nothing about it, at least you have recorded the problems for posterity. Ideally, tag the upstream issue to your GNU Guix patch.
Things get a bit complicated when a build passes in the Keep directory, but fails in GNU Guix. This usually has to do with files being copied into disallowed directories or network access. Just be smart about reading the code and patching it. Worst case you’ll need to build inside a container/VM to find and fix the problems.
./pre-inst-env guix environment erlang -C --ad-hoc erlang vim make git glibc-utf8-locales --no-grafts --pure --share=/tmp/guix-build-elixir-1.5.1.drv-12 --network export LC_ALL=en_US.UTF-8 cd /tmp/guix-build-elixir-1.5.1.drv-12/elixir-1.5.1/
Once the build works you can use standard guix to install the package
./pre-inst-env guix package -i rubyThis will also build from the source tree and blindly merges that directory into your profile, but lacks information for updates etc:
./pre-inst-env guix package -e '(@ (gnu packages ruby) ruby)'guix package -i $(guix build ruby)Where (guix build ruby) is a LISP call which translates into a raw path. With the last example, passing a raw directory name to “guix package -i” does not really know what package it is, so it just blindly merges that directory into your profile. Later upgrades, propagated inputs, and search-path advisories aren’t handled correctly.
One can run:
GUIX_PROFILE=$HOME/.guix-profile . ~/.guix-profile/etc/profile
or
eval `guix package --search-paths`
See http://www.gnu.org/software/guix/manual/html_node/Invoking-guix-package.html.
And nowadays one can also use –search-paths=suffix or –search-paths=prefix, for more flexibility.
debbugs help can be found here.
Bugs can be submitted by E-mail to [email protected] after checking https://debbugs.gnu.org/cgi/pkgreport.cgi?package=guix. Use the parameters as described in bug reporting.
Patches are handled through the GNU debbugs server. A current list can be seen at https://debbugs.gnu.org/cgi/pkgreport.cgi?package=guix-patches or https://bugs.gnu.org/guix-patches. A mailing list is attached you can subscribe to.
In emacs (using guix package -i emacs-debbugs) the same list can be browsed with
M-x debbugs-gnu-search <RET> guix-patches
or
C-u M-x debbugs-gnu <RET> <RET> guix-patches <RET> n y
Possibly you need to add this to your .emacs configuration
(add-to-list 'debbugs-gnu-all-packages "guix-patches")In debbugs mode hit ‘?’ for key-bindings. Use the ‘C’ key from the emacs interface to tag bugs.
To close a bug mail reply and modify the destination address to issuenumber-done@debbugs.
A first time patch is submitted by E-mail to [email protected]. Use the parameters as described in bug reporting.
Each message sent to guix-patches creates a Debbugs entry, as is the case with bug-guix. One can then follow up to [email protected], where NNN is the bug number.
For patch series, please read Glenn’s suggestions. For general questions about Debbugs, see this.
Check the Guix guidelines first. Note that submitting patches is handled via the debbugs interface now, see above section.
To avoid conflictes, before you start, ascertain the Guix tree is at HEAD
git pull guix master git log
Make sure your terminal and editors are running in UTF-8. With vim you can force encoding with
:set bomb :set fileencoding=utf-8 :wq
Use ‘git rebase –interactive’ to merge and squash patches into one. E.g.,
git rebase -i HEAD~4
This can be done with emacs magit. Next use the GNU ChangeLog format which is a header with a filewise change description, for example
gnu: Add Ruby.
* gnu/packages/ruby.scm (Ruby): New file.
* guix/licenses.scm: Add Ruby license information.To change the last commit message do
git commit --amend
Use git format-patch to send a patch to the mailing list.
git format-patch -1to generate a patch file, which you can then send to the Guix debbugs ([email protected]). Note: to generate the last 2 patches use -2.
Create the patch
git format-patch -1 --to [email protected]
Before sending the patch(es) out, make sure tabs are turned into spaces. The emacs commands are here. Lines should be broken (use M-q in emacs). And use the Emacs TAB in guix-prettify-mode to find the right LISP indentation.
Which creates a file 0001-gnu-patchname.patch and mail it with something like
git send-email --from "Pjotr Prins <pjotr.guix@mymail>" --to [email protected] 0001-gnu-patchname.patch
You may also need to install ‘guix package -i git:send-email’ to get E-mail support.
Multiple patches can be passed in with something like
git format-patch -10 # create patches for the past 10 commits git send-email [email protected] *.patch
Probably a good idea to try and send the mail to yourself first. Don’t send the same E-mail twice ;). One example I used
git send-email --from "Pjotr Prins <[email protected]>" --to [email protected] ~/tmp/0001-gnu-ldc-Update-to-1.10.0.patch --suppress-cc=all --subject="[PATCH] Updating ldc to 1.10.0"
To change credentials for the patch use git config. Note that the maintainers will run something like
git am *.patch
to apply the patches.
You can simply reply to the patch with the bug number in the header. So, to resubmit a revised patch with bug number 25704
git send-email --from "Pjotr Prins <pjotr.guix@mymail>" --to [email protected] 0001-gnu-patchname.patch
You can set up an environment to hack on Guix by entering the clone directory and running
guix environment guixThen you can just run make to see if everything builds fine. If it does, make a commit with an appropriate commit message, e.g. by using git rebase (see the guix manual) or by creating a diff between branches (useful when there are conflicts etc.)
git diff master > intermediate.patch
git checkout master
git checkout -b submit_branch
patch -p1 < intermediate.patch
git commit -aNote that the GNU Guix developers want one patch per variable. So submit packages one at a time.
For more information see the official HACKING document in the Guix git repo.
It may be a good idea to keep the master branch in sync with that of Guix. When adding something new checkout a branch first
git checkout -b dev
Now to creat a patch to send to the mailing list do
git commit -a -m 'My last commit' git checkout master git checkout -b submit git rebase --interactive dev
Squash the commits into one
When you write many patches that potentially depend on each other and the review system get choked (the reviewers can’t allways keep up) I resort to a system where I develop patches in a separate branch or even source repository.
If you are using GUIX_PACKAGE_PATH for the separate tree is makes sense to use a different name space (not the gnu directory) and give the packages different names too - so that when you overlap with the GNU Guix package tree there is no name conflict. With the GeneNetwork tree we use the gn/packages path (so modules are in the gn namespace).
The general workflow for adding and maintaining packages is a bit complex. Everything goes via de guix-dev mailing list and includes a review process which can be discouraging and is more geared towards reviewers than towards newbies. This should not discourage you because GNU Guix is great. Note that the reviewers do this work voluntarily and most ‘rules’ have been agreed by the community. In the end your labours will get rewarded. So, how to start?
- Work on a recent git checkout of guix
- Use ‘guix import’ if you can (e.g. for python and R modules)
- Build the package yourself
- If tests are there, make sure they pass
- Test your work with ‘guix lint’
- Create a patch as described above
- Send it to debbugs as described above
- Submit one patch at a time and submit the next one when it goes in
- Be patient, review can take a while - monitor debbugs
With small problems the reviewers will often modify the patch for you. Larger problems you need to fix yourself. See it as a learning process.
Note: sometimes I use an older GNU Guix tree since it is a work in progress and the master may fail for whatever reason. Simply use git cherry-pick to update a single module and it should still work to submit a patch.
Guile has a ‘write’ function which writes to stdout by default. This can be very useful to generate output on package install.
To inspect variables I may inject something like
(write "****************")
(write out)
(write debug)
(newline)
(#f)The last command will compile and break at runtime. Together with the -K option it helps trouble shooting.
If that does not work you can also run a command that fails, such as
(write "HELLO WORLD")
(chdir "END HERE")Another useful hack is to show the environment
(replace 'build
(lambda _
(setenv "LD_LIBRARY_PATH" (getenv "LIBRARY_PATH"))
(invoke "bash" "-c" "set")
(...)
(invoke "./build.sh")))Using -K you can keep the build dir after failure (induced in above paragraph). You may need to patch the source code to make it to work. What I do is use git. Go into the kept directory and run ‘git init’ and add files you change. That way you can generate a patch file that can be added to the guix source tree.
Read the HACKING documentation in the Guix source tree.
There are also videos on hacking in gnu.org/s/guix.
Emacs has powerful support for editing LISP (unsurprisingly, perhaps).
- C-M-f and C-M-b move to forward/backward to matching braces
Send a mail to the bug list, it should look like this:
From: Pjotr Prins <pjotr.public12@email>
To: [email protected]
Bcc:
Subject: guix lint fails with -dc switch missing
Reply-To:
When I run lint on a recent ceckout
./pre-inst-env guix lint
or
./pre-inst-env guix lint python
I get
filtered-port: failed to execute ' -dc ': No such file or directory
Backtrace:
In unknown file:
?: 19 [apply-smob/1 #<catch-closure 16dfcc0>]
In ice-9/boot-9.scm:
63: 18 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
432: 17 [eval # #]When you get the dreaded `server certificate verification failed. CAfile: none CRLfile: none’ you may want to fix the path to certificates. Example:
fatal: unable to access 'https://git.savannah.gnu.org/git/guix.git/': server certificate verification failed. CAfile: none CRLfile: none
If you already have CA certificates, you can point git to them using
the GIT_SSL_CAINFO variable. In .bashrc:
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
It is also possible to checkout a repository using
env GIT_SSL_NO_VERIFY=true git clone URI
(note there are security implications) and next update inside the repo with
git config http.sslVerify false
to override certificate checking.
Occasionally you do a guix pull and regret it. It is pretty easy to recover. Basically a guix pull fetches the latest guix source tree, puts it in the store and symlinks the directory to ~/.config/guix/latest.
So, to change things, change the symlink and point it to a checked out guix git repository (for example).
ls ~/.config/guix/latest gnu gnu.go gnu.scm guix guix.go guix.scm
When working over mobile networks Guix can be painful. The options to check are –no-substites - so you only download source tarballs which can sometimes be less bulky than binaries. Also the –no-grafts option may prevent large downloads of rebuilt (grafted) packages.
The first Ruby gem support by GNU Guix is ruby-i18n (internationalization). The definition looked like
(define-public ruby-i18n
(package
(name "ruby-i18n")
(version "0.6.11")
(source (origin
(method url-fetch)
(uri (string-append "https://github.com/svenfuchs/i18n/archive/v"
version ".tar.gz"))
(sha256
(base32
"1fdhnhh1p5g8vibv44d770z8nq208zrms3m2nswdvr54072y1m6k"))))
(build-system ruby-build-system)
(arguments
'(#:tests? #f)) ; requires bundler
(synopsis "Internationalization library for Ruby")so it downloads the tar ball. The build system looks like
(define ruby-build-system
(build-system
(name 'ruby)
(description "The standard Ruby build system")
(lower lower)))which creates an expression using the standard build-system and the local lower function.
When you install it says
The following environment variable definitions may be needed:
export GEM_PATH="/home/pjotr/.guix-profile/lib/ruby/gems/2.1.3"which contains
ls /home/pjotr/.guix-profile/lib/ruby/gems/2.1.3/gems/i18n-0.6.11/
gemfiles lib MIT-LICENSE README.md testSome packages won’t make it into GNU Guix.
If you have need a special section, simply create a directory with packages and add them to the GUIX_PACKAGE_PATH:
export GUIX_PACKAGE_PATH="~/code/guix-special"
this is also useful for packages that are in Guix but that you would like to customize, for instance with a different set of dependencies or different build flags. Make sure it is a full module, a simple module would be:
(define-module (pylmm)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (gnu packages)
#:use-module (gnu packages python)
#:use-module (guix download)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix utils)
#:use-module (guix build-system gnu)
#:use-module (guix build-system python)
#:use-module (guix build-system trivial)
#:use-module (srfi srfi-1))
(define-public python-pylmm
(package
(name "python-pylmm")
(version "1.0.0")
(source
(origin
(method url-fetch)
(uri (string-append
"https://pypi.python.org/packages/source/p/pylmm/pylmm-"
version ".tar.gz"))
(sha256
(base32 "0bzl9f9g34dlhwf09i3fdv7dqqzf2iq0w7d6c2bafx1nla98qfbh"))))
(build-system python-build-system)
(arguments '(#:tests? #f))
(native-inputs
`(("python-setuptools" ,python-setuptools)))
(home-page "https://github.com/genenetwork/pylmm_gn2")
(synopsis "Python LMM resolver")
(description
"Python LMM resolver")
(license license:gpl-3)))
(define-public python2-pylmm
(package-with-python2 python-pylmm))Save it as a file named pylmm.scm (the name of the module!) and add the path
env GUIX_PACKAGE_PATH=~/python/pylmm_gn2/guix guix package -A python-pylmm python-pylmm 1.0.0 out ~/python/pylmm_gn2/guix/pylmm.scm:15:2
Note that, even though GUIX_PACKAGE_PATH can be a feasible way of adding and maintaining packages, it has two largish downsides: (1) it is removed from the main package tree and therefore not easily shared and integrated and (2) to remain compatible you need to juggle two git trees which may go out of synch.
The Guix daemon contains a build server. It also can distribute built binaries.
See REPRODUCIBLE.org