Skip to content

installation

Jump to bottom
Frank Thomas Tveter edited this page Mar 8, 2022 · 131 revisions

astro-api manual installation documentation

Re-install astro-api
Servers
monitoring

This document contains the relevant console commands for the manual installation of 'astro-api' on an Apache2 webserver on Ubuntu Xenial.

Note that a convenient approach to the installation of astro-api is to use Docker, preferably through a deployment platform like Captainduckduck. For more details see astro-api Docker installation documentation.

Remote login

ssh -X <ubuntu>@<astro.met.no>

Make sure X11 works with sudo 

  xauth list | grep unix`echo $DISPLAY | cut -c10-12` > /tmp/xauth
  sudo xauth add `cat /tmp/xauth` # OK to complain over missing: /root/.Xauthority

  # test that X11 works with sudo
  sudo apt install -y x11-apps
  xclock

Update authorized_keys 

  echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFsLpFTLuy1oFzm+/wVYy4279FKLDfu9D0g9J19iRSIqv2HVcn/flck8MdD2iHapvpPa1s/WqQioXZfBwjfKWRAAfI2dh+8SY5wgp9YwT1GNiuRgcfTP9cZ3g+AR705AAPqKngkMU8m7t/lJVWczeqolASpM11i7awOM8v2qMqGaGXByWJgklDj3EK2ottPOicdi4VcmAntxIqRtkz3dX5gYRrkPCrZGBfdwW4mbwcx2UV2Bda4ZDrSvIm7vbd0G/Ri4QgfZ9nciNSgRjfkDq/GGTZsigyvFkdwKRpsJRafIQMXaIfdqjxQ4rlZL3QbJVTpPodpUke1+38lNc8JZS1 franktt@PC4804" >> ~/.ssh/authorized_keys
  echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuMUx+zg5fIS6OMH4eekjwT1TAMYzaOlLjeGX3+nZOqE4QmDe21eHvdvOIjF+7uiM+pnCRJ2QgcTIgnehX7XIcNDOxW67gYBKyfcPdJMfyE0bU3SCn6ouBXVF2YNaBfmh76o8eJl1AoiFD7LOL/tY5SVHz2NhWazdO6wzV9sNPyp2Oa72jj/uBix/ZHoe2IrbaZNX251qmQXpdfNRk9NLvXQvP6NeAx66zRYfJH/HpzO5kKiW9vu1yyueZE7MIsFfBkZlkk2cb1ADj7T/dop1pkbleib0A1UaF46vnOF7S6H2fc4xAt+SzHJBjhgzjVaBcpgcraHHKrMb3UbbU2jMv franktt@pc4036" >> ~/.ssh/authorized_keys

Install tools

sudo apt -y update

sudo apt-get -y install git tcsh make gfortran devscripts gparted

### you need the following for the pre-relase:
# sudo apt-get install -y libdatetime-perl libxml-libxml-perl

## used by cgi-bin/short.pl
sudo apt-get install -y libxml-libxslt-perl  libdatetime-perl libapache2-request-perl ### libapreq2-dev

sudo cpan XML::XML2JSON   ############### need to press [enter] here

Install emacs (and get rid of the default electric-indentation-mode):

 sudo apt-get install -y emacs

 echo "(when (fboundp 'electric-indent-mode) (electric-indent-mode -1))" >> ~/.emacs

 echo "(set-default 'truncate-lines t)" >> ~/.emacs

Optional, if you will make debian-packages (with self-testing):

sudo apt-get -y install gdebi-core

sudo apt-get -y install debhelper

Generate locale (optional)

This gets rid of annoying "missing locale" warnings.

sudo locale-gen "nb_NO.UTF-8"

# sudo locale-gen "en_US.UTF-8"

Make astro directory

sudo mkdir /astro

sudo chmod a+rwx /astro

Optional, mount a new partition (/dev/vdb1) on /astro.

Create a new partition on your unformatted disk:

sudo gparted

Select your unformatted device (/dev/vdb1) and create the new partition:

  1. device > create partition table -> apply (msdos)
  2. create partition (press "icon with pluss" -> Add)
  3. Apply all operations (press "icon with tick" -> Apply -> Close)
  4. "Gparted menu" -> Quit

Mount /dev/vdb1 on /astro:

sudo su

echo "/dev/vdb1 /astro ext4 defaults 0 0" >>  /etc/fstab

mount /astro

chmod a+rwx /astro

Kill the sudo su shell:

Ctrl + d

Clone astro-api

cd /astro

git clone https://github.com/FrankThomasTveter/astro-api

Compile astro-api

cd astro-api/astro/src;make

cd astro/perl;perl Makefile.PL; make clean; perl Makefile.PL; make

Your installation is local up to this point. The next command will move your files to the system directories.

sudo make install

Opional, install extended JPL ephemeride file (1600-2200)

cd /astro/astro-api/astro/src/asc2eph/work

cp JPLEPH.1600-2200  /astro/astro-api/astro/src/astro/perl/lib/Astro/JPLEPH.405

cd /astro/astro-api/astro/src/astro/perl

sudo make install

Optional, if you want to create and install debian packages (you still need to do sudo make install):

debuild -i -us -uc -b

sudo gdebi -n ../astro-api-perl_0.10-1_amd64.deb

Optional, compile binary programs:

cd /astro/astro-api/astro/src;make bin

The binary programs can be used to create custom JPLEPH.405 files (asc2eph.bin), or test astro-api calculations without using perl or the XML interface (astro.bin).

Uninstall apache2 (optional)

sudo apt-get remove -y --purge  `sudo dpkg --get-selections | grep apache | cut -f 1`

sudo apt autoremove -y

whereis apache2

Install apache2

sudo dpkg --configure -a

sudo apt-get -y install apache2

###sudo apt-get -y install libapache2-request-perl

Move the astro-api html and cgi-bin code to their dedicated apache2 directories (only do this if you are installing apache2):

sudo mv /astro/astro-api/html/astro     /var/www/html  
sudo mv /astro/astro-api/cgi-bin/astro  /var/www/cgi-bin  

chmod a+x /var/www/cgi-bin/astro/short.pl  # temporary bugfix...

Configure apache2 (assuming we will install mod_perl)

sudo mkdir /var/www/cgi-bin  

cd /etc/apache2/

ls

cd sites-enabled/

# emacs 000-default.conf
sudo perl -pi -w -e 's|DocumentRoot \/var\/www\/html
|DocumentRoot \/var\/www\/html
        ScriptAlias \"\/astro\/cgi-bin\/\" \"\/var\/www\/cgi-bin\/astro\/\"
        <Directory \"\/var\/www\/cgi-bin\/astro\/\">
           AllowOverride None
           Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
           PerlOptions +ParseHeaders
           Require all granted
        </Directory>

        #### Allow CORS-access (access from any client): 
        Header set Access-Control-Allow-Origin \"*\"
|g;' 000-default.conf


# save and exit emacs:
# Ctrl+s  Ctrl+x


### Add "Header" module if you are using CORS access: 
cd /etc/apache2/mods-available/; sudo a2enmod headers

sudo ln -s /etc/apache2/mods-available/cgid.load /etc/apache2/mods-enabled/

sudo ln -s /etc/apache2/mods-available/cgid.conf /etc/apache2/mods-enabled/

#ls -l /etc/apache2/mods-enabled/ | grep cgi

sudo apachectl -t  

tail -f /var/log/apache2/error.log

Ctrl + c

Optional, for optimal performance install mod_perl 

sudo apt-get -y install libapache2-mod-perl2

# sudo apt install aptitude

# sudo aptitude install libapache2-mod-perl2

Optional, configure mod_perl 

### Add mod_perl

sudo su
cat << EOF >> /etc/apache2/conf-available/mod_perl.conf 

PerlSwitches -w
PerlSwitches -T

Alias /astro/cgi-bin/ /var/www/cgi-bin/astro/
<Directory /var/www/cgi-bin/astro/>
    AddHandler perl-script .cgi .pl
    PerlResponseHandler ModPerl::PerlRun
    PerlOptions +ParseHeaders
    Options +ExecCGI
</Directory>

<Location /perl-status>
    SetHandler perl-script
    PerlResponseHandler Apache2::Status
    Require ip 127.0.0.1 10.0.0.0/24
</Location>
EOF

a2enconf mod_perl  

# make mod_perl work for code that is not thread-safe (just in case)
a2dismod mpm_event
a2enmod mpm_prefork

# Finally load mod_perl in apache2...
# # a2enmod : =Apache2 ENable MODule
# sudo a2enmod perl apreq2 # <- this is the recommended way, but I have not tested it...
# This works, but is not recommended:
 cd /etc/apache2/mods-enabled
 ln -sf ../mods-available/perl.load .
 ln -sf ../mods-available/apreq2.load .

Kill the sudo su shell:

Ctrl + d

(Re)start apache2 webserver

sudo service apache2 restart

The apache2 webserver should now be running and available under your localhost.

Test installation

Compare your local demonstration of the astro-api to the online demonstration. Debug if necessary.

Check apache2 error-messages:

tail -f /var/log/apache2/error.log  
Ctrl+c

Check access to "dummy" webpage:

 sudo apt-get -y install firefox  
 firefox localhost/astro/cgi-bin/test.pl

Check browser client errors in Firefox Web-Konsole: Ctrl+shift+K

Run server-scripts offline

cd /var/www/cgi-bin/astro            # apache2 directory
# cd /astro/astro-api/cgi-bin/astro  # use this instead if the server-scripts were not moved to the apache2 directory

./event
./state
./offline.pl '?debug=1&eventVal1=59.417084&eventVal2=10.483212900000012&eventVal3=0&eventStart=2017-11-12T16%3A11%3A43.108Z&eventSearch=0&event1Id=600'

Note that you do NOT have to install apache2/mod_perl to run the server-scripts offline.

Compile new JPL ephemeride files

cd /astro/astro-api/astro/src/asc2eph/work
# download the new ephemeride files that cover your desired time-span
wget ftp://ssd.jpl.nasa.gov/pub/eph/planets/ascii/de405/ascp1990.405
# ...or download all the available ephemeride files (Year 1600 - 2200)
wget ftp://ssd.jpl.nasa.gov/pub/eph/planets/ascii/de405/ascp\*
./run.pl   # generate the new ephemeride file

Install new JPL ephemeride files

cd /astro/astro-api/astro/src/asc2eph/work

cp JPLEPH  /astro/astro-api/astro/src/astro/perl/lib/Astro/JPLEPH.405

### cp JPLEPH.1600-2200  /astro/astro-api/astro/src/astro/perl/lib/Astro/JPLEPH.405

cd  /astro/astro-api/astro/src/astro/perl

make clean;perl Makefile.PL;make;sudo make install

sudo service apache2 restart

Upgrade Ubuntu

sudo apt-get update          # Fetches the list of available updates
sudo apt-get upgrade -y      # Strictly upgrades the current packages
sudo apt-get dist-upgrade -y # Installs updates (new ones)

Re-install astro-api

cd /astro

rm -rf astro-api.old; mv astro-api astro-api.old

git clone https://github.com/FrankThomasTveter/astro-api 

cd astro-api/astro/src;make

cd astro/perl;perl Makefile.PL; make clean; perl Makefile.PL; make

cp /astro/astro-api/astro/src/asc2eph/work/JPLEPH.1600-2200  /astro/astro-api/astro/src/astro/perl/lib/Astro/JPLEPH.405

sudo make install

sudo rm -rf /var/www/html/astro;    sudo mv /astro/astro-api/html/astro     /var/www/html

sudo rm -rf /var/www/cgi-bin/astro; sudo mv /astro/astro-api/cgi-bin/astro  /var/www/cgi-bin  

sudo apache2ctl graceful # do not loose requests in production

#  sudo service apache2 restart

Adding response-time to apache2 access logs

Edit /etc/apache2/apache2.conf and add %D in the "LogFormat" strings (usually the "combined" format).

sudo perl -pi -w -e 's|\"" combined|\" %D" combined|g;' /etc/apache2/apache2.conf

Log rotate (delete old logs)

Changes to /etc/logrotate.d/apache2

 apt-get install logrotate

 # daily => hourly
 sudo perl -pi -w -e 's|daily|hourly|g;' /etc/logrotate.d/apache2
 # rotate 14 => rotate 2
 sudo perl -pi -w -e 's|rotate 14|rotate 2|g;' /etc/logrotate.d/apache2

Alternative

  sudo cat << EOF > /etc/logrotate.d/apache2
  /var/log/apache2/*.log {
    hourly
    missingok
    rotate 2
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
            if /etc/init.d/apache2 status > /dev/null ; then \
                /etc/init.d/apache2 reload > /dev/null; \
            fi;
    endscript
    prerotate
            if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                    run-parts /etc/logrotate.d/httpd-prerotate; \
            fi; \
    endscript
  }
  EOF

Finally run these commands:

 sudo /usr/sbin/logrotate -d /etc/logrotate.d/apache2 # check
 sudo /usr/sbin/logrotate -f /etc/logrotate.d/apache2 # run

 sudo cp /etc/cron.daily/logrotate /etc/cron.hourly/
 sudo systemctl restart cron

Logrotate will now keep apache2 logs for a maximum of 2 hours.

Local installation

If you can not run root commands (sudo), and have to run the scripts offline from the command line, you must install astro-api locally. Your commands are then typically:

git clone https://github.com/FrankThomasTveter/astro-api 

cd astro-api/astro/src;make

cd astro/perl;perl Makefile.PL; make clean; perl Makefile.PL; make

To execute your offline.pl script, perl needs to know where the astro-api system files can be found:

   Astro/Api.pm 
   Astro/Astro.pm 
   auto/Astro/Astro.so
   Astro/JPLEPH.405

The JPLEPH.405 file must be located together with Astro.pm. Use perl -V to see the default perl library search directories that perl will search for these files. You may specify additional perl library search directories using a use lib statement in your script, for instance:

  #!/usr/bin/perl
  #
  use 5.006;
  use lib '/tmp/build_ee1ebcc2df7a4ead15b8692674cde431/astro-api/astro/src/astro/perl/blib';
  use Astro::Astro;
  use strict;
  use warnings;
  use Carp qw(croak);
  my @lines=Astro::Astro::pm_event(shift);
  foreach my $line (@lines) {
      print "$line\n";
  }

Several use lib statements can be used. Perl expects to find the directory Astro/, that contains the file Astro.pm, in one of the perl library search directories.

Notes on local gfortran installation

If you have a local gfortran installation, perl will need to know the path to the gfortran library file, typically named libgfortran.a or libgfortran.so, before it creates the astro-api object. If astro-api is compiled without this information, you will typically get a warning of the type

  cd astro/perl;perl Makefile.PL; make clean; perl Makefile.PL; make
  ...
  Warning (mostly harmless): No library found for -lgfortran 
  ...

If you ignore this warning and try to execute the offline.pl script, perl will abort saying

  Astro.so: undefined symbol: _gfortran_st_open

To fix this you should try to specify the directory where the gfortran library resides in a -L/full/path/ statement in the LIBS variable in the astro-api/astro/src/astro/perl/Makefile.PL file before re-compiling astro-api in astro-api/astro/src/astro/perl:

  make clean; perl Makefile.PL; make

Location of astro-api files

In a standard astro-api installation, the perl library search directories are

  perl -V
  ...
  @INC:
  /etc/perl
  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1
  /usr/local/share/perl/5.22.1
  /usr/lib/x86_64-linux-gnu/perl5/5.22
  /usr/share/perl5
  /usr/lib/x86_64-linux-gnu/perl/5.22
  /usr/share/perl/5.22
  /usr/local/lib/site_perl
  /usr/lib/x86_64-linux-gnu/perl-base
  .

The astro-api system files are located at:

  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Astro/Astro.pm
  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/auto/Astro/Astro.so
  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Astro/Api.pm
  /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Astro/JPLEPH.405

Note that astro-api system files are here located in the perl library search directory /usr/local/lib/x86_64-linux-gnu/perl/5.22.1.

Servers

1.0 servers

Old servers

Datarom-A:

Datarom-B:

Old test:

Datarom-A:

Datarom-B:

Demo (astro.met.no/astro):

Test:

Monitoring

Prometheus

The following receipt is used to install the latest node_exporter on each astro-api node. Make sure you are using bash shell. #!/bin/bash -xe

Create the working directory.

  mkdir ~/node_exporter
  cd ~/node_exporter

Start super-user shell.

  sudo su

Remove old version of node_exporter (optional).

  apt-get remove  -y prometheus-node-exporter;

Create service user.

  #Create User
  adduser --no-create-home --disabled-login --shell /bin/false --gecos "Node Exporter User" node_exporter

Download the latest node_exporter and install it.

  node_exporter_version=1.0.1

  # Download Binary
  wget https://github.com/prometheus/node_exporter/releases/download/v$node_exporter_version/node_exporter-$node_exporter_version.linux-amd64.tar.gz

  # Untar 
  tar xvzf node_exporter-$node_exporter_version.linux-amd64.tar.gz

  # Copy Exporter
  cp node_exporter-$node_exporter_version.linux-amd64/node_exporter /usr/local/bin/

  # Assign Ownership Again
  chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create service

# Creating Service File
cat <<- EOF > /etc/systemd/system/node_exporter.service    
[Unit]
  Description=Node Exporter
  Wants=network-online.target
  After=network-online.target
[Service]
  User=node_exporter
  Group=node_exporter
  Type=simple
  ExecStart=/usr/local/bin/node_exporter
[Install]
  WantedBy=multi-user.target
EOF

# Reload and Start
export LC_ALL=en_US.UTF-8; 
export LANG=en_US.UTF-8; 
systemctl daemon-reload; 
systemctl enable node_exporter
systemctl start node_exporter
systemctl status node_exporter

Kill the sudo su shell:

Ctrl + d

Check if installation works:

  curl localhost:9100/metrics
  sudo service node_exporter status

Stop and start service.

  sudo service node_exporter stop
  sudo service node_exporter start

Sensu monitoring (obsolete)

sensu may be used to monitor the CPU-load on your astro-api server.

Install sensu software on your astro-api server:

  sudo su
  apt-get -y install ruby ruby-dev
  gem install sensu-plugins-load-checks
  gem install sensu-plugins-http

  wget -O - http://repositories.sensuapp.org/apt/pubkey.gpg | apt-key add
  echo deb http://repositories.sensuapp.org/apt sensu main > /etc/apt/sources.list.d/sensu.list
  apt-get -y update
  apt-get -y install sensu

Edit sensu client configuration on your astro-api server

  cat /etc/sensu/conf.d/client_config.json
  cat << EOF > /etc/sensu/conf.d/client_config.json
  {
      "client": {
      "name": "astro.met.no",
      "address": "157.249.177.142",
      "subscriptions": ["mydash", "mysystem"],
      "keepalive": {"thresholds": {"warning": 60, "critical": 120}, "handlers": ["mailer"]},
      "redact": [
        "password",
        "pass",
        "api_key"
      ]
    },
    "rabbitmq": [
      {
        "host": "sensu.int.met.no",
        "port": 5671,
        "user": "",
        "password": "",
        "vhost": "",
        "heartbeat": 30,
        "prefetch": 50,
        "ssl": {    
          "cert_chain_file": "/etc/sensu/ssl/rabbit-clients-cert.pem",
          "private_key_file": "/etc/sensu/ssl/rabbit-clients-key.pem"
        }
      },
      {
        "host": "sensu-backup.int.met.no",
        "port": 5671,
        "user": "",
        "password": "",
        "vhost": "",
        "heartbeat": 30,
         "prefetch": 50,
       "ssl": {    
          "cert_chain_file": "/etc/sensu/ssl/rabbit-clients-cert.pem",
          "private_key_file": "/etc/sensu/ssl/rabbit-clients-key.pem"
         }
      }
    ]
  }
  EOF

Edit the empty user/password/vhost fields, according to the secrets provided by the sensu-server.

  emacs /etc/sensu/conf.d/client_config.json

Remember to update server certificates:

mkdir -p /etc/sensu/ssl
cd /etc/sensu/ssl/
wget https://gitlab.met.no/infra/sensu-client/raw/master/files/rabbit-clients-cert.pem?inline=false -O rabbit-clients-cert.pem
wget https://gitlab.met.no/infra/sensu-client/raw/master/files/rabbit-clients-key.pem?inline=false -O rabbit-clients-key.pem

Add cpu-load and web-server check on your astro-api server:

  cat /etc/sensu/conf.d/check-cpu-load.json
  cat << EOF > /etc/sensu/conf.d/check-cpu-load.json
  {
    "checks": {
      "cpu-load": {
        "command": "check-load.rb  -c 100,95,90 -w 90,85,80",
        "standalone": true,
        "handlers": ["mailer"],
        "interval": 60,
        "occurrences": 1
      }
     }
  }
  EOF

  cat << EOF > /etc/sensu/conf.d/check-http.json
  {
    "checks": {
      "web-server": {
        "command": "check-http.rb -u http://localhost/astro/index.html",
        "standalone": true,
        "handlers": ["mailer"],
        "interval": 60,
        "occurrences": 1
      }
    }
  }
  EOF

  /etc/init.d/sensu-client restart

Enable sensu client on system boot

  sudo update-rc.d sensu-client defaults
  sudo update-rc.d sensu-client enable
  #sudo update-rc.d sensu-client disable

Start/stop client

  sudo service sensu-client start
  sudo service sensu-client stop

Maintenance

LogFormat (log access times)

  cd /etc/apache2/conf-enabled/
  ln -s /etc/apache2/conf-available/other-vhosts-access-log.conf .

  cat /etc/apache2/apache2.conf
  LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
  LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D" combined
  LogFormat "%h %l %u %t \"%r\" %>s %O" common
  LogFormat "%{Referer}i -> %U" referer
  LogFormat "%{User-agent}i" agent

event.pl

80-VISIBLE MOON

100-INITIAL MOON STATE

110-MOON POSITION

120-INITIAL SUN STATE

130-SUN POSITION

140-VISIBLE SUN

150-SOLSTICE

190-EARTH ORBIT

200-MOON ORBIT

210-MOON PHASE

260-MOON ILLUMINATION

290-LUNASTICE

300-MERCURY

340-VENUS

380-MARS

420-JUPITER

460-SATURN

500-TRANSIT

550-LUNAR ECLIPSE

560-SUN RISE

640-TWILIGHT

750-SUN TIME

800-MOON RISE

900-POLAR SUN

940-POLAR MOON

980-SOLAR ECLIPSE

1000-SOLAR SYSTEM POSITIONS

Clone this wiki locally