Skip to content

ESGF Upgrade

Jump to bottom Edit New page
Prashanth Dwarakanath edited this page Feb 14, 2018 · 75 revisions

Upgrade Procedure

Unix User

You must be root or effectively root to run this program, prefixing the command with sudo will not allow the use of needed environment variables! If you must use sudo, do so only to become root proper then source your user's .[bash]rc file so that root has it's environment set accordingly! Or you can more simply become root using sudo's "-s" flag.

Stop the Node

cd /usr/local/bin
esg-node stop

Clean the webapps directory

Please do this compulsorily, in order to have consistent behavior upon upgrading.

rm -fr /usr/local/tomcat/webapps/*
rm -fr /usr/local/src

Download and Execute the Bootstrap Script

To upgrade to a 'devel' install

cd /usr/local/bin
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/devel/2.6/4/esgf-installer/esg-bootstrap --no-check-certificate  
chmod 555 esg-bootstrap  
./esg-bootstrap --devel

To upgrade to a 'master' install

wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/2.6/5/esgf-installer/esg-bootstrap --no-check-certificate  
chmod 555 esg-bootstrap  
./esg-bootstrap

Check Installer Script Version

esg-node --version
Version: v2.6.5-master-release
Release: Asgard
Earth Systems Grid Federation (http://esgf.llnl.gov)
ESGF Node Installation Script

Start the installation process

script -c 'esg-node --upgrade' upgrade.log

Primary Configuration

Please select the ESGF distribution mirror for this installation (fastest to slowest): 
-------------------------------------------
 [1] http://dist.ceda.ac.uk/esgf 
 [2] http://esg-dn2.nsc.liu.se/esgf 
 [3] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf 
 [4] http://aims1.llnl.gov/esgf 
-------------------------------------------

Are you ready to begin the installation? [Y/n] Y

Some notes on choices during the upgrade process

You may see questions related to adding of replica shards. If prompted whether you'd like to add a NEW replica shard for port 8983/8984, please answer 'Y'.

If you plan to publish CMIP6 data make sure you answer Y to the question regarding that (not a default).

Most sites should automatically "peer with supernode" (this is Node Manager module specific). Exceptions are the several Tier 1 sites yet to install their supernode (answer N) which should run n the index node at LLNL, JPL, CEDA, DKRZ, and IPSL.

Tier 1 IDP only - We encourage all sites to deploy the SLCS service so please answer Y.

Expert Options

  • You can specify a local mirror for use, by specifying the --use-local-mirror option. You'll be then prompted interactively for a mirror url.

  • Due to any transient network/system-availability issue, if you are not able to download the bootstrap from the primary mirror, you can obtain them from other mirrors i.e. CEDA, LIU, or AIMS. The bootstrap scripts specific to the mirrors are called esg-bootstrap.ceda, esg-bootstrap.liu, and esg-bootstrap.aims respectively. Note that you have to save them with their original names, as their checksums differ. For instance, to download the bootstrap from AIMS, and execute it, you would do the following:

    wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/2.6/5/esgf-installer/esg-bootstrap --no-check-certificate
    chmod 555 esg-bootstrap.aims
    ./esg-bootstrap.aims

Post-upgrade

  • Please restart the node after upgrading: esg-node restart

  • Recommended to run esg-node --install-local-certs if you have your signed certs in place for gridftp/myproxy as the certificates might be replaced by temporary ones (they last only 1 month and some services will be unavailable without clear message), as frequently observed to happen after upgrades. (TODO when this is fixed in the installer, we can remove this manual step)

  • An (additional) localhost shard may have be written to esgf_shards_static.xml. This will result in datasets listed twice in the facet counts.

  • If your installation/upgrade is NOT related to the Data Challenge 2 (DC2) of CDNOT, please downgrade esgprep to version 2.7.37, like this:

[root@esg-idx ~]# source /usr/local/conda/bin/activate esgf-pub
(esgf-pub) [root@esg-idx ~]#
(esgf-pub) [root@esg-idx ~]# pip install esgprep===2.7.35
Collecting esgprep===2.7.35
  Downloading esgprep-2.7.35.tar.gz
Requirement already satisfied: lockfile>=0.12.2 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: esgconfigparser>=0.1.7 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: requests>=2.17.3 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: fuzzywuzzy>=0.15.0 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: netCDF4<=1.3.1,>=1.2.0 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: hurry.filesize>=0.9 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: treelib<=1.4.0,>=1.3.5 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: tqdm>=4.11.2 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from esgprep===2.7.35)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from requests>=2.17.3->esgprep===2.7.35)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from requests>=2.17.3->esgprep===2.7.35)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from requests>=2.17.3->esgprep===2.7.35)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from requests>=2.17.3->esgprep===2.7.35)
Requirement already satisfied: numpy>=1.7 in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from netCDF4<=1.3.1,>=1.2.0->esgprep===2.7.35)
Requirement already satisfied: setuptools in /usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages (from hurry.filesize>=0.9->esgprep===2.7.35)
Building wheels for collected packages: esgprep
  Running setup.py bdist_wheel for esgprep ... done
  Stored in directory: /root/.cache/pip/wheels/87/db/59/6e341941532d56a2e3918d8f8530c917d6d9fc9bd000f1d72b
Successfully built esgprep
Installing collected packages: esgprep
  Found existing installation: esgprep 2.7.37
    Uninstalling esgprep-2.7.37:
      Successfully uninstalled esgprep-2.7.37
Successfully installed esgprep-2.7.35
Add a custom footer
Add a custom sidebar
Clone this wiki locally