- Introduction
- Current state
- Package Nokogiri
- Creating the gem-derived package
- Getting rid of Bundler
- Continuous integration (CI)
- The final nokogiri package description
- Notes
This document explains how to go from creating a software package to creating a non-native software packaging system in GNU Guix. Examples of such systems are Python pip, emacs ELPA packages and (here) Ruby gems.
Ruby developers have multiple needs when running a system when it comes to Ruby versions and modules (called gems). GNU Guix has great support for versioning and control of the dependency graph. Here we discuss gem support in GNU Guix (this is a work in progress).
Basically we are aware of the following use cases after installing a version of Ruby (say 2.1.6):
- A user/sysadmin wants to install a gem in the GNU Guix store
- A user wants to install a gem using rubygems outside GNU Guix and local to $HOME
- A developer wants to use bundler which installs gems in the source tree for development and testing
In all cases, when Ruby code needs to include a file, resolving the location of the gem should start in 3, next 2, then 1. As an example of supporting a complex gem we use the example of Nokogiri which has over 1,000 installation related questions on Stack Overflow.
GNU Guix has native gem support through the rubygems-uri procedure and ruby-build-system!
That means you can create a simple package description which can install a native gem:
(define-public ruby-json
(package
(name "ruby-json")
(version "1.8.3")
(source
(origin
(method url-fetch)
(uri (rubygems-uri "json" version))
(sha256
(base32
"1nsby6ry8l9xg3yw4adlhk2pnc7i0h0rznvcss4vk3v74qg0k8lc"))))
(build-system ruby-build-system)
(arguments '(#:tests? #f)) ; dependency cycle with sdoc
(synopsis "JSON library for Ruby")
(description "This Ruby library provides a JSON implementation written as
a native C extension.")
(home-page "http://json-jruby.rubyforge.org/")
(license (list license:ruby license:gpl2)))) ; GPL2 onlyMore examples can be found here
The rest of this document describes the discovery path of building the nokogiri software through GNU Guix. The final package description is fairly simple!
GNU Guix now build gems from a tar-ball using a Ruby build environment. Installing ruby-hoe
guix package -i ruby-hoe
creates a symlink to the executable sow in
/home/pjotrp/.guix-profile/bin/sow -> /gnu/store/ccx2ri5l8y6qbrzkw4p0gq8l647kxn0g-ruby-hoe-3.13.1/bin/sow
and the libraries are available through the symlink in
$HOME/.guix-profile/lib/ruby/gems/2.2.0/gems/hoe-3.13.1/ -> /gnu/store/ccx2ri5l8y6qbrzkw4p0gq8l647kxn0g-ruby-hoe-3.13.1/lib/ruby/gems/2.2.0/gems/hoe-3.13.1
Running sow means you’ll need to add the GNU Guix gem locations to the GEM_PATH, e.g.,
env GEM_PATH=~/.guix-profile/lib/ruby/gems/2.2.0 sow
At this point the GEM_PATH is not set automatically. Guix prints the PATH after installing a package so you can add it to you environment. You can view the recommended paths for your profile with
guix package --search-paths
For end-users we may want to automate that.
To see the actual paths and libraries that a Ruby uses, check
guix environment ruby-2.1 --search-paths
Currently, to find gems, you have to set the GEM_PATH. I use a script for this.
The gem tool comes with the GNU Guix Ruby installation. This means you can run gem after tweaking the local PATHs. An example can be found here. Run it as
. ruby-guix-env
Note: it also sets the path for GNU Guix global gems.
Native gems include some C or C++ code.
Currently the GNU Guix gem tool does build native gems. See below the infamous Nokogiri example.
Bundler works in GNU Guix, both installed though Guix and through gems. See the Nokogiri example below. I prefer, however, not to use bundler at all. GNU Guix makes bundler obsolete. See the section on that below.
GNU Guix provides a number of native gems. On my system I use
bigdecimal (1.2.6)
hoe (3.13.1)
i18n (0.6.11)
io-console (0.4.3)
json (1.8.1)
log4r (1.1.10)
minitest (5.4.3)
nokogiri (1.6.6.2)
power_assert (0.2.2)
psych (2.0.8)
rake (10.4.2)
rake-compiler (0.9.5)
rdoc (4.2.0)
rspec (3.2.0)
rspec-core (3.2.3)
rspec-expectations (3.2.1)
rspec-mocks (3.2.1)
rspec-support (3.2.2)
test-unit (3.0.8)Note the notoriously difficult to support nokogiri gem. It was added through the exercise below.
The local gem installer (after setting paths) installs other gems, including the natively building cucumber gems. E.g.
builder (3.2.2)
cucumber (2.0.2)
cucumber-core (1.2.0)
diff-lcs (1.2.5)
gherkin (2.12.2)
io-console (0.4.3)
json (1.8.1)
multi_json (1.11.2)
multi_test (0.1.2)Nokogiri is a great test case because the gem contains both Ruby and C files (that need to be compiled into shared library nokogiri.so) and it depends on external C libraries which are not always on a system (libxml2 and libxslt, for example).
But first a tip: because the build is reproducible you don’t have to fix a package in one go. Personally, I like to work incrementally at fixing dependencies. One at a time. GNU Guix always stops where you were last.
First using the script I set up the Guix environment so it looks like
set|grep guix
ACLOCAL_PATH=$HOME/.guix-profile/share/aclocal
BASH=$HOME/.guix-profile/bin/bash
CPATH=$HOME/.guix-profile/include
GEM_PATH=$HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0:$HOME/.guix-profile/lib/ruby/gems/2.1.0/
GUILE_LOAD_COMPILED_PATH=$HOME/.guix-profile/share/guile/site/2.0
GUILE_LOAD_PATH=$HOME/.guix-profile/share/guile/site/2.0
LIBRARY_PATH=$HOME/.guix-profile/lib
LOCPATH=$HOME/.guix-profile/lib/locale
PATH=$HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0/bin:$HOME/.guix-profile/bin:$HOME/.guix-profile/sbin:/usr/bin:/bin
PKG_CONFIG_PATH=$HOME/.guix-profile/lib/pkgconfigNote the PATH still contains /usr/bin for convenience, though you should be able to do without.
Trying a naive
gem install nokogiri
results in ‘ERROR: Failed to build gem native extension’ due to a failing libxml2 build. Nokogiri packages its own version of libxml2 and tries to compile that. One way to solve this error is by fixing the compile problem (the logs say it is libtool related), the other way is to install libxml2 in Guix and tell nokogiri where to find it.
guix package -i libxml2
Guix symlinks both static and shared libraries in ~/.guix-profile/lib/ so we tell nokogiri where to find them
gem install nokogiri -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xml2-lib=$HOME/.guix-profile/lib
First Nokogiri complains ‘libxml2 version 2.9.2 or later is highly recommended, but proceeding anyway’ - we can fix that later. Next error is missing libxslt, but now we know what to do
guix package -i libxslt
and
gem install nokogiri -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib
and now the build succeeded
nokogiri -v # Nokogiri (1.6.6.2)
That was rather easy for a notoriously difficult gem! And it looks like we can formalize this in a Guix package. Note that I cheated a little. Since /usr/bin is still in the path I am (probably) still using some of the build tools of the underlying distribution (running gem on its own does not isolate the build). I could aim to fix that, but it will come out when we add a proper guix package anyway (guix builds are fully isolated).
One interesting check is to see what the nokogiri.so shared library that we built links against, e.g.
ldd $HOME/.gem/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6/2.1.0/extensions/x86_64-linux/2.1.0-static/nokogiri-1.6.6.2/nokogiri/nokogiri.so
and validate all the paths are pointing at the GNU Guix store. You don’t want to mix in libraries that are non-guix because it suggests things are missing. Note that the Nokigiri documentation also suggests gem path options for
--with-iconv-dir=/path/to/dir --with-zlib-dir=/path/to/dir
as well as
--with-exslt-dir=/path/to/dir --with-exslt-config=/path/to/exslt-config.
But none of these were needed here.
Running bundler naively
bundle install
results in the same library issues with ‘Gem::Ext::BuildError: ERROR: Failed to build gem native extension’. Bundler also needs to be told where to find the libraries.
The first try was to configure bundler by adding to .bundle/config
BUNDLE_BUILD__NOKOGIRI: "--use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib"
Unfortunately, this does not work as it does not prevent bundler for starting to build the libxml2. This should not happen with the –use-system-libraries option. To check the bundler setting see
bundle config build.nokogiri
:
Set for your local app (app/.bundle/config): "--use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib"
But somehow these do not get honoured by extconf.rb. After reading the source and some trying inside the build dir I found the environment variable
~/.gems/bundler/ruby/2.1.0/gems/nokogiri-1.6.1/ext/nokogiri$ env NOKOGIRI_USE_SYSTEM_LIBRARIES=1 ruby extconf.rb --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib
resulted in
/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory
predefs is part of the GNU C library (libc6), so it is perhaps strange it does not get picked up (well, Guix even isolates away the native system - go the Guix gcc compiler does not see /usr/include). Adding –with-opt-include=/usr/include/x86_64-linux-gnu does find it. Added that to bundler’s config
BUNDLE_PATH: $HOME/.gems/bundler/ BUNDLE_DISABLE_SHARED_GEMS: '1' BUNDLE_BUILD__NOKOGIRI: " --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=/usr/include/x86_64-linux-gnu"
and ran
env NOKOGIRI_USE_SYSTEM_LIBRARIES=1 bundle
and the thing builds. Better even, also Cucumber builds and all the test pass for bio-vcf (the tool I want to ultimately package).
You may want to check config settings with
bundle config
Note we should have used predefs.h from the store glibc-2.21/include/stdc-predef.h instead. It looks like Nokogiri is using an older include. This suggests what needs to be done:
error: #error "Never use <bits/predefs.h> directly; include <stdc-predef.h> instead."
Now we now how gem/bundler builds Nokogiri we have a chance at building the package from source and bundling it into GNU Guix. The tar ball can be found on https://github.com/sparklemotion/nokogiri/releases.
Unpack the tar ball and extconf.rb builds with
cd ext\nokogiri env LD_LIBRARY_PATH=$HOME/.guix-profile/lib LIBRARY_PATH=$HOME/.guix-profile/lib \ NOKOGIRI_USE_SYSTEM_LIBRARIES=1 ruby extconf.rb \ --with-xml2-include=$HOME/.guix-profile/include/libxml2 \ --with-xslt-include=$HOME/.guix-profile/include/libxslt \ --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib \ --with-opt-include=/usr/include/x86_64-linux-gnu \ --with-opt-include=$HOME/.guix-profile/include
and make
env LIBRARY_PATH=$HOME/.guix-profile/lib make
check the linked paths
ldd nokogiri.so
linux-vdso.so.1 (0x00007ffc9f3e1000)
libexslt.so.0 => $HOME/.guix-profile/lib/libexslt.so.0 (0x00007fb6c45aa000)
libxslt.so.1 => $HOME/.guix-profile/lib/libxslt.so.1 (0x00007fb6c436b000)
libxml2.so.2 => $HOME/.guix-profile/lib/libxml2.so.2 (0x00007fb6c4006000)
libpthread.so.0 => $HOME/.guix-profile/lib/libpthread.so.0 (0x00007fb6c3de9000)
libdl.so.2 => $HOME/.guix-profile/lib/libdl.so.2 (0x00007fb6c3be4000)
libcrypt.so.1 => $HOME/.guix-profile/lib/libcrypt.so.1 (0x00007fb6c39ad000)
libm.so.6 => $HOME/.guix-profile/lib/libm.so.6 (0x00007fb6c36ab000)
libc.so.6 => $HOME/.guix-profile/lib/libc.so.6 (0x00007fb6c330a000)
libgcc_s.so.1 => /gnu/store/76afr0pfbnimz7rdad35y5yd753myjhk-gcc-4.9.2-lib/lib/libgcc_s.so.1 (0x00007fb6c30f4000)
liblzma.so.5 => /gnu/store/h86jd7lyd6lny3yz30d44gi4b0mz73in-xz-5.0.4/lib/liblzma.so.5 (0x00007fb6c2ed1000)
libz.so.1 => /gnu/store/yx7c449ds3psyrn40h4nfvsb7xqqzziy-zlib-1.2.7/lib/libz.so.1 (0x00007fb6c2cb8000)
libgcrypt.so.20 => /gnu/store/r16v30hlw2d6n232rm37p53qy8rpr7f2-libgcrypt-1.6.3/lib/libgcrypt.so.20 (0x00007fb6c29db000)
libgpg-error.so.0 => /gnu/store/63lp72xz64axrbrlvpyln449v42h0zbh-libgpg-error-1.18/lib/libgpg-error.so.0 (0x00007fb6c27ca000)
/gnu/store/wiqbxcvzj3r35hd55yxzz919b1dv1hnv-glibc-2.21/lib/ld-linux-x86-64.so.2 (0x00007fb6c49de000)as it should be - though with the GNU Guix package the .guix-profile’s will point to proper store locations.
nokogiri.so is the C-part of the gem. The Ruby part sits in ./bin and ./lib in the tarball. These can simply be copied into the GEM_HOME. But reading the current implementation of the GNU Guix ruby-build-system, it creates a gem first using a gemspec
rake gem:spec
create the gem
gem build nokogiri.gemspec
install using our earlier trick
env C_INCLUDE_PATH=$HOME/.guix-profile/include gem install --local nokogiri-1.6.6.2.20150629081149.gem -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=$HOME/.guix-profile/include
which (now) fails with
38:26: fatal error: linux/limits.h: No such file or directory #include <linux/limits.h>
Actually, this is not so bad. The environment gets picked up in a GNU Guix package, so let’s move on. The install path (mostly) works.
In the next step we start with an existing GNU Guix package so we can just fill in the missing pieces. First I synchronized the Guix source and checked out a new branch named nokogiri
git pull --recurse-submodules guix master git checkout -b nokogiri
now we need to make sure the environment is correct (as described in ./HACKING.org)
make
make sure gnutls is installed
guix package -i gnutls guix download https://github.com/sparklemotion/nokogiri/archive/v1.6.6.2.tar.gz
which gives
/gnu/store/v2hc2imgzgar4srfh64svkvas4ha07xz-v1.6.6.2.tar.gz 1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j
Then I copied an existing package from gnu/packages/ruby.scm and started filling in
(define-public ruby-nokogiri
(package
(name "ruby-nokogiri")
(version "1.6.6.2")
(source (origin
(method url-fetch)
(uri (string-append
"https://github.com/sparklemotion/nokogiri/archive/v"
version ".tar.gz"))
(file-name (string-append name "-" version ".tar.gz"))
(sha256
(base32
"1dpmmxr8azbyvhhmw9hpyk3dds577vsd6c312gh2s7kgjd98nd9j"))))
(build-system ruby-build-system)
(arguments
`(#:tests? #f)) ; no test suite
(synopsis "Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser")
(description "Nokogiri parses and searches XML/HTML very quickly, and also has correctly implemented CSS3 selector support as well as XPath 1.0 support.")
(home-page "http://www.nokogiri.org/")
(license license:expat)))Note the MIT license is also known as the X11 or expat license.
Now we have the package let’s see if it is there
./pre-inst-env guix package -A ruby-nokogiri ruby-nokogiri 1.6.6.2 out gnu/packages/ruby.scm:504:2
now build it
./pre-inst-env guix package -K -i ruby-nokogiri
the -K switch will keep the unpacked build directory. The first error pops up
ERROR: No files matching pattern: "\\.gemspec$"
which makes sense, because earlier we had to run first
rake gem:spec
The builder says that it kept build directory `/tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-0’. So in a different terminal do
cd /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-0 . environment-variables
and you are at the state of the error (with environment). Running
rake gem:spec
it complains Gem::LoadError: Could not find ‘hoe’ (>= 0) among 9 total gem(s). It is interesting to note that the build is completely isolated from the rest of the system, so any dependencies not explicitly added will fail. To check run
set
And when you do add it, it will be visible to the package forever and this is key to the build being reproducible.
Note: you may need to change the permissions of the build directory to try stuff by hand. As root
chown user -R /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-*
We have to add the hoe dependency first.
(native-inputs
`(("ruby-hoe" ,ruby-hoe)))
and retry the build. Now the new build is in /tmp/nix-build-ruby-nokogiri-1.6.6.2.drv-1. This way we keep reiterating until the package works. One of the interesting errors was LoadError: cannot load such file – rake/extensioncompiler since we had not seen that earlier. That is part of the rake-compiler gem. The gems are listed in the Rakefile as
["hoe-bundler", ">= 1.1"],
["hoe-debugging", "~> 1.2.0"],
["hoe-gemspec", ">= 1.0"],
["hoe-git", ">= 1.4"],
["minitest", "~> 2.2.2"],
["rake", ">= 0.9"],
["rake-compiler", "~> 0.9.2"],
["racc", ">= 1.4.6"],
["rexical", ">= 1.0.5"]We have to add the necessary missing package(s) to GNU Guix. The rake-compiler package becomes something like:
(define-public ruby-rake-compiler
(package
(name "ruby-rake-compiler")
(version "0.9.5")
(source (origin
(method url-fetch)
(uri (string-append
"https://github.com/rake-compiler/rake-compiler/archive/v"
version ".tar.gz"))
(file-name (string-append name "-" version ".tar.gz"))
(sha256
(base32
"07lk1vl0jqcaqwjjhmg0qshqwcxdyr5kscc9xxm13m03835xgpf3"))))
(build-system ruby-build-system)
(arguments
'(#:tests? #f ; needs cucumber
#:phases (modify-phases %standard-phases
(add-before 'build 'remove-cucumber-rake-task
(lambda _
;; Remove cucumber test file because the
;; dependencies are not available right now.
(delete-file "tasks/cucumber.rake")))
(replace 'build
(lambda _ (zero? (system* "rake" "gem")))))))
(synopsis "Building and packaging helper for Ruby native extensions")
(description "Rake-compiler proivides a framework for building and
packaging native C and Java extensions in Ruby.")
(home-page "https://github.com/rake-compiler/rake-compiler")
(license license:expat)))
Note it needs to remove tasks/cucumber.rake to prevent those tasks from running. Also we override the build system because this package runs
rake gem
to create the gem instead of the default ‘gem build $package.gemspec’ as defined in ./guix/build/ruby-build-system.scm. After successfully installing that package we simply add the dependency to the nokogiri package.
Next phase the build complains that Errno::ENOENT: No such file or directory @ rb_sysopen - ports/archives/libxml2-2.9.2.tar.gz. This is because Nokogiri build wants to find the source and patch libxml2 for itself. Previously we used rake gem:spec, but that is no longer available with this later version, check rake task options with
rake -T
Running the Nokogiri build with
rake newb
results in a mini_portile (LoadError). Now mini-portile we don’t need (it is another packaging system). But I found out that running
rake gem
twice will generate the gem. Now the install phase fails on mini-portile. That means we need to replicate the earlier gem install command with its switches. First we need to add libxml2 and libxslt as dependencies.
This requires adding at the top of ruby.scm
#:use-module (gnu packages xml)
and inside the nokogiri package definition
(inputs
`(("zlib" ,zlib)
("libxml2" ,libxml2)
("libxslt" ,libxslt)))
resulting in a build with
The following files will be downloaded: /gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2 /gnu/store/sprxqr56hm7p9wcy17bm2vj99k1mr779-libxslt-1.1.28
Nice. The build fails (of course), but now inside the build directory you can find the settings. E.g.
cat environment-variables
shows build variables, such as
export LIBRARY_PATH=(...):/gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2/lib
where dependencies can be found. To reference such a dependency you can add variables in the package. One example could be
(let ((libxml2 (assoc-ref inputs "libxml2")) do something
So the earlier –with-xml2-include switch can become something like
(string-append "--with-xml2-include=" libxml2 "/include/libxml2")
Note: At some point the guile REPL may come in handy to see what is happening. See the guix-notes HACKING guide for more information.
It is interesting to see what other packages implement. The ruby-git package adds an absolute path for the git binary with
(arguments
'(#:phases (modify-phases %standard-phases
(add-before 'build 'patch-git-binary
(lambda* (#:key inputs #:allow-other-keys)
;; Make the default git binary an absolute path to the
;; store.
(let ((git (string-append (assoc-ref inputs "git")
"/bin/git")))
(substitute* '("lib/git/config.rb")
(("'git'")
(string-append "'" git "'")))
;; Fix a test that expects the binary to be simply
;; 'git'.
(substitute* '("tests/units/test_logger.rb")
(("def test_logger")
(string-append
"def test_logger\n"
"Git::Base.config.binary_path = 'git'")))
#t)))
(add-before 'check 'create-fake-home
(lambda _
;; The test suite runs 'git config --global' commands,
;; so a fake home directory is needed for them to
;; succeed.
(let ((fake-home (string-append (getcwd) "/fake-home")))
(mkdir fake-home)
(setenv "HOME" fake-home)))))))After patching out the ‘mini_portile’ dependency from the Rakefile and adapting the gem install –local the next error was ERROR: While executing gem … (Gem::FilePermissionError) You don’t have write permissions for the /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/gems/2.2.0 directory.
Great! We do not want the library installed inside Ruby, but in its own store. Thanks GNU Guix for pointing that out! We need to override the install-dir.
Inside the build directory the following worked after disabling the libxml2 check in extconf.rb (so another patch is required).
gem install --install-dir /tmp --local pkg/nokogiri-1.6.6.2.gem -- --use-system-libraries --with-xml2-include=/gnu/store/s4vwk3f0ainazh2czl5k5gsainpiby6i-libxml2-2.9.2/include/libxml2
Note that the libxml2 include file is in a non-standard place, so it needs to be defined. Even so, less configuration is needed than the earlier build-by-hand exercise. Remember we had to specify
env C_INCLUDE_PATH=$HOME/.guix-profile/include gem install --local nokogiri-1.6.6.2.20150629081149.gem -- --use-system-libraries --with-xml2-include=$HOME/.guix-profile/include/libxml2 --with-xslt-include=$HOME/.guix-profile/include/libxslt --with-xml2-lib=$HOME/.guix-profile/lib --with-xslt-lib=$HOME/.guix-profile/lib --with-opt-include=$HOME/.guix-profile/include
GNU Guix’ input variable resolves the standard library and include paths. Rather then using /tmp, we also use the targetdir out.
Next thing we know the ruby-nokogiri package installs!
After installing Nokogiri
./pre-inst-env guix package -i ruby-nokogiri The following package will be upgraded: ruby-nokogiri 1.6.6.2 → 1.6.6.2 /gnu/store/ynwfr9mfs5w3xhbxn1sgbcqrq0mh4gdx-ruby-nokogiri-1.6.6.2
the binary complains
$HOME/.guix-profile/bin/nokogiri
/gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'nokogiri' (>= 0) among 9 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=$HOME/.gem/ruby/2.2.0:/gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/gems/2.2.0', execute `gem env` for more information
from /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/dependency.rb:324:in `to_spec'
from /gnu/store/9iifw37m8vd5bkj0fh67ndc5f2da46wb-ruby-2.2.2/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
from $HOME/.guix-profile/bin/nokogiri:22:in `<main>'
which means the nokogiri module is not found. Doing a
find ~/.guix-profile/lib/ruby/ -name nokogiri*
finds nothing. That is disappointing. The binary nokogiri wants to run the module as a gem from a GEM_PATH. In the store we find
/gnu/store/ynwfr9mfs5w3xhbxn1sgbcqrq0mh4gdx-ruby-nokogiri-1.6.6.2/gems/nokogiri-1.6.6.2/lib/nokogiri.rb
On my underlying Debian system the gem path is:
selinunte:~$ gem env
RubyGems Environment:
- RUBYGEMS VERSION: 1.8.23
- RUBY VERSION: 1.9.3 (2012-04-20 patchlevel 194) [x86_64-linux]
- INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
- RUBY EXECUTABLE: /usr/bin/ruby1.9.1
- EXECUTABLE DIRECTORY: /usr/local/bin
- GEM PATHS:
- /var/lib/gems/1.9.1
- $HOME/.gem/ruby/1.9.1which means we need a system-wide path for gems - which does not exist in Guix. The Guix way is to create symlinks in a profile (usually ./guix-profile/), so the logical thing is to either symlink gnu/store/ynwfr(…)-ruby-nokogiri-1.6.6.2/gems/nokogiri-1.6.6.2/lib onto ~.guix-profile/lib/ruby/2.2.2/ or symlink a gems directory and add ~/.guix-profile/lib/gems/2.2.2/ to the GEM_PATH. Since this is clearly a gem path, I favour the latter. So we have to add that gem support to GNU Guix.
When I look into a standard bundler install it has the identical shared library 3x in
./bundler/ruby/2.1.0/gems/nokogiri-1.6.1/lib/nokogiri/nokogiri.so ./bundler/ruby/2.1.0/gems/nokogiri-1.6.1/ext/nokogiri/nokogiri.so ./bundler/ruby/2.1.0/extensions/x86_64-linux/2.1.0-static/nokogiri-1.6.1/nokogiri/nokogiri.so
not sure why that is - probably an artifact of nokogiri’s build system (can we now state it is a mess?). Only the first one is probably required.
Interestingly I find gems (and other directories) are symlinked in ./guix-profile! But they are at the root of the profile. To fix this all we need to do is ‘hoist’ the relevant directories inside the package into ./lib/gems/#{version}.
The ruby-rspec-core package does that. And reading the ruby-build system it does that (we overrode that with ruby-nokogiri):
(define* (install #:key source inputs outputs #:allow-other-keys)
(let* ((ruby-version
(match:substring (string-match "ruby-(.*)\\.[0-9]$"
(assoc-ref inputs "ruby"))
1))
(out (assoc-ref outputs "out"))
(gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0")))
(setenv "GEM_HOME" gem-home)
(mkdir-p gem-home)
(zero? (system* "gem" "install" "--local"
(first-matching-file "\\.gem$")
;; Executables should go into /bin, not /lib/ruby/gems.
"--bindir" (string-append out "/bin")))))So, rather than overriding the install phase, we would be better of adding the one option it introduces for finding the libxml2 include! You can see the install phase has #:allow-other-keys, so we can modify the ruby-build-system to allow for an option we can name #:gem-install-option. The xpdf package does something similar
#:phases
(alist-replace
'install
(lambda* (#:key outputs inputs #:allow-other-keys #:rest args)
(let* ((install (assoc-ref %standard-phases 'install))
(out (assoc-ref outputs "out"))
(xpdfrc (string-append out "/etc/xpdfrc"))
(gs-fonts (assoc-ref inputs "gs-fonts")))
(apply install args)
(substitute* xpdfrc
(("/usr/local/share/ghostscript/fonts")
(string-append gs-fonts "/share/fonts/type1/ghostscript"))
(("#fontFile") "fontFile"))))
%standard-phases)))Note the ‘apply install’ calling back into the build-system.
Fixing it for Nokogiri is also a simplification because the underlying install method is used. We pass in the extra gem-flags parameter.
(alist-replace
'install
(lambda* (#:key inputs outputs #:allow-other-keys #:rest args)
(let* ((out (assoc-ref outputs "out"))
(libxml2 (assoc-ref inputs "libxml2"))
(gem-flags (string-append
"--use-system-libraries --with-xml2-include="
libxml2 "/include/libxml2"))
(install (assoc-ref %standard-phases 'install)))
(apply install #:gem-flags gem-flags args)))Now we can install Nokogiri and run it after the GEM_PATH is set:
export GEM_PATH=$HOME/.guix-profile/lib/ruby/gems/2.2.0/
nokogiri
Nokogiri: an HTML, XML, SAX, and Reader parser
Usage: nokogiri <uri|path> [options]
Examples:
nokogiri http://www.ruby-lang.org/
nokogiri ./public/index.html
curl -s http://nokogiri.org | nokogiri -e'p $_.css("h1").length'
Options:
--type type Parse as type: xml or html (default: auto)
-C file Specifies initialization file to load (default $HOME/.nokogirirc)
-E, --encoding encoding Read as encoding (default: none)
-e command Specifies script from command-line.
--rng <uri|path> Validate using this rng file.
-?, --help Show this message
-v, --version Show version
Success!!
After sending the patch around I got feedback from others. Most importantly, rather than overriding the install phase I should be using arguments.
One interesting point of note is that nokogiri is using ruby 2.2.2 while I have ruby 2.1.6 in my profile. We’ll look into that later because we don’t want to mix the two. The resolution will be similar to that of Dave’s ‘gem-with-ruby’ procedure hanging around somewhere that does something like:
(define-public ruby1.9-nokogiri
(gem-with-ruby ruby-nokogiri ruby-1.9))It recursively overrides the Ruby version used for packages that use the Ruby build system.
In the next phase we are going to modify the build system. In the Nokogiri exercise we were working from a downloaded tar-ball not native to rubygems.
Because nokogiri is such a complex beast, let’s try first with a simpler gem that has only a few dependencies.
Log4r is a long standing popular Ruby gem with few dependencies. Doing a simple
gem install log4r
on my GNU Guix ruby install rapidly with
Fetching: log4r-1.1.10.gem (100%) Successfully installed log4r-1.1.10 Parsing documentation for log4r-1.1.10 Installing ri documentation for log4r-1.1.10 Done installing documentation for log4r after 0 seconds
That looks promising. Now we want to make it into a proper GNU Guix package for posterity. The command line steps
wget https://rubygems.org/downloads/log4r-1.1.10.gem gem install --ignore-dependencies --install-dir ~/tmp/gems log4r-1.1.10.gem
With Guix the destination will be ~/.guix-profile/lib/ruby/gems/2.2.0 or similar.
The ignore-dependencies switch is important because it prevents gem from fetching dependencies on its own. The GNU Guix rubygem packages we are going to create should take care of that.
Basically I copied the existing ruby-build-system to facilitate gem installs. I’ll probably have to make one later again, but this is a good strategy for testing new things.
Bundler aims to solve a problem that Guix solves - i.e. dependencies. The good thing about bundler is that it creates an environment which can be replicated for continuous integration (Travis CI). But it is not as robust as Guix can do - still, and a surprising amount of tooling depends on bundler. Checking an average bundler managed Gemfile of one of my projects it lists over 70 lines of dependencies. Seriously, that is the full gem dependency graph (but not the actual graph because it does not list lower dependencies, such as the Ruby interpreter itself and glibc, which are indeed part of the more robust Guix graph).
The whole Ruby developer community uses bundler. With Guix, however, we have profiles and bundler can and should be thrown out.
To get rid of bundler in an existing project, remove the .bundler dir, the Gemfile and Gemfile.lock and remove bundler binary from the path.
gem uninstall bundler guix package -r bundler (maybe ruby-bundler later)
Make sure it is gone
gem list which bundle bundle command not found
Next start removing bundler dependencies in the files. I had some in my test initialisation - make sure it is removed from all sources! Note, to get decent errors you may want to (temporarily) remove the exception catching in the cucumber source gems/cucumber-2.0.2/lib/cucumber/cli/main.rb.
After the fixes I could run cucumber without bundler (without the cucumber modification)! Make a note of the gems you need to pull in by ‘hand’.
Annoyingly bundler gets pulled in by some other gems (jeweler for one). So every time you run gem install locally make sure it is not there.
Note at any time you can remove the local gems and run gem list to see what Guix gives us:
rm -rf ~/.gem/p3vzqwxavyfchwjw2bxnq365sr1ap99b-ruby-2.2.2/ gem list
without jeweler, bundler and dependencies the list became quite a bit shorter.
To test Rubygems in Travis with bundler, simply provide the Gemfile. Alternatively pull in depending gems separately with gem in the .travis.yml file:
install: - gem install cucumber rspec regressiontest
This has the advantage of being faster and purer. Even better go the Guix way:
Guix has a build farm with CI. It even provides build logs, e.g., http://hydra.gnu.org/build/573030/log/raw
(define-public ruby-nokogiri
(package
(name "ruby-nokogiri")
(version "1.6.6.2")
(source (origin
(method url-fetch)
(uri (rubygems-uri "nokogiri" version))
(sha256
(base32
"1j4qv32qjh67dcrc1yy1h8sqjnny8siyy4s44awla8d6jk361h30"))))
(build-system ruby-build-system)
(arguments
;; Tests fail because Nokogiri can only test with an installed extension,
;; and also because many test framework dependencies are missing.
'(#:tests? #f
#:gem-flags (list "--" "--use-system-libraries"
(string-append "--with-xml2-include="
(assoc-ref %build-inputs "libxml2")
"/include/libxml2" ))))
(native-inputs
`(("ruby-hoe" ,ruby-hoe)
("ruby-rake-compiler", ruby-rake-compiler)))
(inputs
`(("zlib" ,zlib)
("libxml2" ,libxml2)
("libxslt" ,libxslt)))
(propagated-inputs
`(("ruby-mini-portile" ,ruby-mini-portile)))
(synopsis "HTML, XML, SAX, and Reader parser for Ruby")
(description "Nokogiri (鋸) parses and searches XML/HTML, and features
both CSS3 selector and XPath 1.0 support.")
(home-page "http://www.nokogiri.org/")
(license license:expat)))The latest version you can find in git.
It says so in the ./bundle/config file.
Nokogiri lists many solutions here. We should add ours.