-
Notifications
You must be signed in to change notification settings - Fork 21
ESGF Installation From scratch
Regardless of whether you have installed and administered an ESGF node previously, please read the following document on ESGF policies, as this should influence what type of installation you should do:
http://esgf.llnl.gov/media/pdf/ESGF-Policies-and-Guidelines-V1.0.pdf
We have migrated the ESGF installation procedure to use Ansible. Please see the following:
Information on the use of esgf-installer scripts are below, though maintenance of these scripts have been discontinued on April 1, 2019.
Currently, no devel releases of ESGF are available.
2.8.2-master-release
ESGF has an auto-installer which is capable of automating installs for all known valid combinations of roles (all, data, data+compute+index,...), and you are strongly advised to use the auto-installer, in order to eliminate problems due to due to errant keystrokes etc. You can find the page providing details about the autoinstaller here. If this doesn't work for you, or if you prefer to do an expert-install, follow the documentation given below.
Platform
RHEL6; CentOS6; Scientific Linux 6
Selinux
selinux needs to be set to Disabled. This can be done by ensuring that the line SELINUX=disabled is present and uncommented in the /etc/sysconfig/selinux file. To confirm, reboot the machine and run getenforce; the output should be Disabled.
Unix User
You must be root or effectively root to run this program, prefixing the command with sudo will not allow the use of needed environment variables! If you must use sudo, do so only to become root proper then source your user's .[bash]rc file so that root has its environment set accordingly! Or you can more simply become root using sudo's "-s" flag.
Prerequisites
- Ports to open: 80, 443, (GridFTP: 2811, 50000-51000), (MyProxy: 7512)
- Outgoing connections: 7512 RabbitMQ 5671 or 32272
- During the installation process, you'll be asked for a globus username and password. You'll need to sign up for a globusonline account beforehand, and provide the username and password when asked for it. You can sign up for an account at https://www.globus.org/SignUp
One way to clear previous installations is via the esg-purge.sh script. For example, to do a full uninstall (don't forget to exit the root shell afterwards in order to drop remaining ESGF-related environment variables and functions, before starting the new install):
cd /usr/local/bin
source /usr/local/bin/esg-purge.sh && esg-purge all && rm -f /etc/yum.repos.d/esgf.repo
Download and Execute the Bootstrap Script
First of all:
script -a install.log # don't forget to issue exit at the end of the installation.
cd /usr/local/bin
To setup the latest 'devel' install
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/devel/2.8/0/esgf-installer/esg-bootstrap --no-check-certificate
chmod 555 esg-bootstrap
./esg-bootstrap --devel
To setup the latest 'master' install
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/2.8/2/esgf-installer/esg-bootstrap --no-check-certificate
chmod 555 esg-bootstrap
./esg-bootstrap
Check Installer Script Version
esg-node --version
Version: v2.8.2-master-release
Release: Winterfell
Earth Systems Grid Federation (http://esgf.llnl.gov)
ESGF Node Installation Script
Start the installation process
esg-node --set-type [data|compute|idp|index|all]
esg-node --install
Use script instead of tee to log the installation. tee can cause configuration files corruption. Make sure you specify the type without square bracket or pipe.
Please select the ESGF distribution mirror for this installation (fastest to slowest):
-------------------------------------------
[1] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf
[2] http://dist.ceda.ac.uk/esgf
[3] http://esg-dn2.nsc.liu.se/esgf
[4] http://aims1.llnl.gov/esgf
-------------------------------------------
Some other notes
If you plan to publish CMIP6 data make sure you answer Y to the question regarding that (not a default).
As we are refactoring the node manager, the "peer with "supernode" question is no longer relevant. Everyone can answer 'N' to the question, but it would not be harmful if the question is skipped.
Tier 1 IDP only - We encourage all sites to deploy the SLCS service so please answer Y.
Post installation
- Required to restart the node after installing: esg-node restart
Unix User
You must be root or effectively root to run this program, prefixing the command with sudo will not allow the use of needed environment variables! If you must use sudo, do so only to become root proper then source your user's .[bash]rc file so that root has it's environment set accordingly! Or you can more simply become root using sudo's "-s" flag.
Stop the Node
cd /usr/local/bin
esg-node stop
Clean the webapps directory
Please do this compulsorily, in order to have consistent behavior upon upgrading.
rm -fr /usr/local/tomcat/webapps/*
rm -fr /usr/local/src
Download and Execute the Bootstrap Script
To upgrade to a 'devel' install
cd /usr/local/bin
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/devel/2.8/0/esgf-installer/esg-bootstrap --no-check-certificate
chmod 555 esg-bootstrap
./esg-bootstrap --devel
To upgrade to a 'master' install
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/2.8/2/esgf-installer/esg-bootstrap --no-check-certificate
chmod 555 esg-bootstrap
./esg-bootstrap
Check Installer Script Version
esg-node --version
Version: v2.8.2-master-release
Release: Winterfell
Earth Systems Grid Federation (http://esgf.llnl.gov)
ESGF Node Installation Script
Start the installation process
script -c 'esg-node --upgrade' upgrade.log
Primary Configuration
Please select the ESGF distribution mirror for this installation (fastest to slowest):
-------------------------------------------
[1] http://dist.ceda.ac.uk/esgf
[2] http://esg-dn2.nsc.liu.se/esgf
[3] http://distrib-coffee.ipsl.jussieu.fr/pub/esgf
[4] http://aims1.llnl.gov/esgf
-------------------------------------------
Are you ready to begin the installation? [Y/n] Y
Additional Notes
-
If you plan to publish CMIP6 data make sure you answer Y to the question regarding that (not a default).
-
Tier 1 IDP only - We encourage all sites to deploy the SLCS service so please answer Y.
Expert Options
- You can specify a local mirror for use, by specifying the --use-local-mirror option. You'll be then prompted interactively for a mirror url.
- Due to any transient network/system-availability issue, if you are not able to download the bootstrap from the primary mirror, you can obtain them from other mirrors i.e. CEDA, LIU, or AIMS. The bootstrap scripts specific to the mirrors are called esg-bootstrap.ceda, esg-bootstrap.liu, and esg-bootstrap.aims respectively. Note that you have to save them with their original names, as their checksums differ. For instance, to download the bootstrap from AIMS, and execute it, you would do the following:
wget -O esg-bootstrap http://distrib-coffee.ipsl.jussieu.fr/pub/esgf/dist/2.8/2/esgf-installer/esg-bootstrap --no-check-certificate
chmod 555 esg-bootstrap.aims
./esg-bootstrap.aims
Post-installation
For self-generated certificates, install them like this:
(Note: Do not worry about warnings/errors during the node stop)
./esg-node stop
./esg-node --install-keypair /etc/tempcerts/hostcert.pem /etc/tempcerts/hostkey.pem
For the CA chain question, use /etc/tempcerts/cacert.pem
./esg-node start
If you have signed certificates, use those instead:
./esg-node stop
./esg-node --install-local-certs
# Only if you don't have a commercial/third-party web-container certificate (eg Verisign, DigiCert, LetsEncrypt)
# Use those in this step instead.
./esg-node --install-keypair /etc/esgfcerts/hostcert.pem /etc/esgfcerts/hostkey.pem
If your certificates were signed by:
The IPSL CA, use /etc/grid-security/certificates/0597e90c.0.
The NSC CA, use /etc/grid-security/certificates/cd6ccc41.0.
The ANL CA, use /etc/grid-security/certificates/99eb76fc.0.
If you have a web-container certificate provided by a commercial entity (see above) you will install those using those.
./esg-node --install-keypair <cert-file> <key-file>
Your provider should have provided you with the CA chain. Additionally contact IWT to ensure that the CA root cert is in the ESGF truststore.
When prompted, restart the node with ./esg-node restart.