Skip to content

Release process for Bioconductor docker images Release 3.15

Jump to bottom
Nitesh Turaga edited this page Apr 26, 2022 · 1 revision

Release process of bioconductor_docker images

Notes to capture the release process of the bioconductor/bioconductor_docker image for RELEASE_3_15.

General

The image is based on rocker/rstudio:4.2.0, which is based on ubuntu 20.04

The rocker project has dockerfiles in the rocker-versioned2.

Although the latest Dockerfiles are available, the images under rocker's dockerhub repo are missing. To look at the Dockerfile(s), follow the link https://github.com/rocker-org/rocker-versioned2/tree/master/dockerfiles

Steps to update devel

  1. Before making any changes to the master branch, create a RELEASE_3_15 branch with

     	 git branch RELEASE_3_15

  2. In the Dockerfile: Update version number of BIOCONDUCTOR_DOCKER_VERSION to the latest X.Y.Z, where X.Y represents the Bioconductor version of devel.

    • E.g For Bioconductor 3.16, the BIOCONDUCTOR_DOCKER_VERSION will be 3.16.0.

  3. Change the install.R file to reflect the latest verison of Bioconductor in BiocManager::install(version='devel'). This depends on the current release cycle of Bioconductor. Check with the team if there are questions.

  4. Try to rebuild the image with,

       docker build -t bioconductor/bioconductor_docker:devel

    There were a few issues with the system libraries,

    • The following libraries do not INSTALL in the main apt-get install block, but install in the subsequent block. This might be because of missing dependencies. But this is something I have to look further into.
      • libmariadb-dev-compat
      • libjpeg62-dev

  5. Make sure the docker-compose.yml file has the correct values as well in both places.

  6. Validity check: The final step is installing all the packages and to triage which packages DO NOT install on the new devel image.

Steps to update RELEASE_3_15

  1. Checkout the RELEASE_3_15 branch,

     `git checkout RELEASE_3_15`

  2. The BIOCONDUCTOR_DOCKER_VERSION number of the branch just gets incremented in the Z by 1. Since it is the same Bioc version as the previous devel.

  3. Make sure the correct rocker/rstudio: is being used. If there are doubts about this check the http://bioconductor.org/checkResults/devel/bioc-LATEST/ (devel) and http://bioconductor.org/checkResults/release/bioc-LATEST/ (release) versions of R on the build machines. They should match.

  4. Try to rebuild the image with

        docker build -t bioconductor/bioconductor_docker:RELEASE_3_15 .

    There were a few issues with the system libraries, (same as the above with devel

    • The following libraries do not INSTALL in the main apt-get install block, but install in the subsequent block. This might be because of missing dependencies. But this is something I have to look further into.
      • libmariadb-dev-compat
      • libjpeg62-dev

  5. There are no changes in the install.R file, except to install BiocManager 3.15

  6. Remove the following lines in the Dockerfile, i.e, no devel build variables in the RELEASE branch

      # DEVEL: Add sys env variables to the DEVEL image
  RUN curl -O http://bioconductor.org/checkResults/devel/bioc-LATEST/Renviron.bioc \
 		 && cat Renviron.bioc | grep -o '^[^#]*' | sed 's/export //g' >>/etc/environment \
 		 && cat Renviron.bioc >> /usr/local/lib/R/etc/Renviron.site \
 		 && rm -rf Renviron.bioc

  7. Validity check: Install packages on the new image and triage which packages DO NOT install. Set option(Ncpus=<more than default>) for faster installation.

Create singularity images as the final step.

  1. In the newly created RELEASE_X_Y branch, rename the file Singularity to Singularity.RELEASE_X_Y.

  2. Inside the file, change the lines, From: bioconductor/bioconductor_docker:devel to From: bioconductor/bioconductor_docker:RELEASE_X_Y. Below is example for RELEASE_3_11.

       Bootstrap: docker
   From: bioconductor/bioconductor_docker:RELEASE_3_15

Code to test installation of all packages

The following code should be run to install all packages (running on a machine with 16 cores)

options(Ncpus=14)

installed <- rownames(installed.packages())
biocsoft <- available.packages(repos = BiocManager::repositories()[["BioCsoft"]])

## Packages which failed to install on docker image
to_install <- rownames(biocsoft)[!rownames(biocsoft) %in% installed]

BiocManager::install(to_install)

Addendum - Github Actions

  1. Github actions should be removed from the newly created RELEASE_X_Y branch. Dockerhub does the builds for the RELEASE_X_Y branches as they are "stable". Push to the image with an updated BIOCONDUCTOR_DOCKER_VERSION number will trigger build to the RELEASE_X_Y branch.

  2. Under the current structure, the way we build the image for the bioconductor/bioconductor_docker:devel image is via Github actions. The action pulls the rocker/r-ver:<tag> image and the rocker:rstudio:<tag>, builds both of those images on an instance, then builds the bioconductor/bioconductor_docker:devel image on the latest version of those build images.

    • The devel image is updated weekly at 3pm on Friday.

    • To replace with the rockerdev image stack, we need to be able to get the github repos and Dockerfiles of the latest rockerdev/r-ver and rockerdev/rstudio images build on Ubuntu 18.04.

  3. TODO: This github action needs to be edited to reflect the latest changes through rocker.

  4. Validity check: To check validity of the weekly build, it'll be useful to temporarily set the CRON job on the scheduled github action to run every 3 hours to debug if needed and change back to the weekly cycle once it is done.

Failing packages

Clone this wiki locally