Skip to content

FTS3 Cluster

Jump to bottom
dweinholz edited this page Sep 18, 2023 · 1 revision

Requirements

Setting up the FTS3 cluster needs at least three machines:

  1. A database instance where the MySQL DB is installed.
  2. A proxy machine with NGINX.
  3. At least one FTS3 node machine.

x.509 client and server certificates are required. If you do not have an authority to create these, you may find a guide here.

Credentials to an S3 bucket are required. Further, the bucket needs to be configured.

A CentOS 7 base image is required.

Official fts3 docs can be found here.

Setting up a database instance

This installation guide for MySQL 8 is mostly taken from here with additions from here.

Start a VM with CentOS 7 and SSH to it.
All following commands need to be run with sudo or as root user, to ease the process run as root user: sudo su.
Update the machine with yum update and reboot it. Next, run the following commands:

rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
yum --enablerepo=mysql80-community install mysql-community-server
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
service mysqld start
grep "A temporary password" /var/log/mysqld.log

Save the shown password as it is the temporary root password. Next, change it to something sensible with

mysql_secure_installation

and yes to all prompted questions. Restart MySQL with service mysqld restart and set autostart with chkconfig mysqld on.
Create the FTS3 database by logging into mysql with mysql -u root -p and run CREATE DATABASE DATABASE_NAME;.

MySQL is now installed, but we need the schema and software from FTS3. Install wget with yum install wget and run

wget https://fts-repo.web.cern.ch/fts-repo/fts3-prod-el7.repo
wget https://dmc-repo.web.cern.ch/dmc-repo/dmc-el7.repo
mv fts3-prod-el7.repo /etc/yum.repos.d/
mv dmc-el7.repo /etc/yum.repos.d/
yum update
yum install epel-release
yum install centos-release-scl-rh
yum update
yum install fts-mysql

Now we import the schema, cd /usr/share/fts-mysql/ and mysql -u root -p DATABASE_NAME < fts-schema-8.0.1.sql. Next, we create the user the cluster needs to interact with the database. Log into the database with mysql -u root -p and run

CREATE USER 'A_USER_NAME'@'%' IDENTIFIED WITH mysql_native_password BY 'A_GOOD_PASSWORD';
GRANT ALL ON DATABASE_NAME.* TO 'A_USER_NAME'@'%';
FLUSH PRIVILEGES;
GRANT SUPER ON *.* to 'A_USER_NAME'@'%';
FLUSH PRIVILEGES;

Now, we need to insert the first user into the database who will be able to configure FTS3 with the REST API. Again, log into the MySQL DB with mysql -u root -p DATABASE_NAME or with mysql -u root -p and run USE DATABASE_NAME;. Next, you will need the subject line from the client certificate that you want to use (the information can be viewed with openssl x509 -noout -text -in client.cert.pem).
When your subject is something like this: C=DE, ST=SomeProvince, L=SomeCity, O=SomeOrganization, CN=A_USER/[email protected], add the user like this insert into t_authz_dn (dn, operation) values ('/C=DE/ST=SomeProvince/L=SomeCity/O=SomeOrganization/CN=A_USER/[email protected]', 'config');. This is the DN of the client.

Last, set a security group in OpenStack to open port 3306. Only the FTS3 nodes need access which is why 192.168.0.0/16 is sufficient.

Setting up an FTS3 node

This setup is the same for every FTS3 node. To save time, start one FTS3 node, configure it and create a Snapshot from which multiple nodes can be spawned.

Start an instance with CentOS 7 as base image and SSH into it. To save the sudo commands, run as root user with sudo su. Update your machine with yum update and reboot it. Next install all required fts-software by running:

yum install wget
wget https://fts-repo.web.cern.ch/fts-repo/fts3-prod-el7.repo
wget https://dmc-repo.web.cern.ch/dmc-repo/dmc-el7.repo
wget https://fts-repo.web.cern.ch/fts-repo/fts3-depend-el7.repo
mv fts3-prod-el7.repo /etc/yum.repos.d/
mv dmc-el7.repo /etc/yum.repos.d/
mv fts3-depend-el7.repo /etc/yum.repos.d/
yum update
yum install epel-release
yum install centos-release-scl-rh
yum update
yum install fts-server
yum install fts-rest-server
yum install fts-monitoring
yum install fts-server-selinux fts-rest-server-selinux fts-monitoring-selinux
yum install fts-mysql

Next, configure the node with two files /etc/fts3/fts3config and /etc/fts3/fts3restconfig. Both need the same changes.

  • dbusername=A_USER_NAME
  • dbpassword=A_GOOD_PASSWORD
  • dbconnectstring=IP_OF_THE_DATABASE_INSTANCE/DATABASE_NAME # E.g. 192.168.1.155/fts3
  • infosys=false
  • sitename=NAME_OF_CLOUD_CENTER # Will be shown in the web monitoring

Now place the server certificate at /etc/grid-security/hostcert.pem with root:root as owner and 0444 permissions, the server key at /etc/grid-security/hostkey.pem with root:root as owner and 0400 permissions and the certificate authority chain at /etc/grid-security/certificates/CA.pem with 0444 permissions.

FTS3 can only stream from HTTP/S to S3, in /etc/gfal2.d/http_plugin.conf change the line to DEFAULT_COPY_MODE=streamed.
In /etc/httpd/conf.d/fts3rest.conf remove the SSLCARevocationPath, SSLCARevocationCheck and SSLCACertificatePath lines and instead add SSLCACertificateFile /etc/grid-security/certificates/CA.pem.
Now you can start the Node with

systemctl start fts-server
systemctl start fts-qos
systemctl start httpd

Last, add a security group in OpenStack that opens ports 8449 for the NGINX proxy and 8446 for everyone.

Setting up an NGINX proxy

Star an instance with Ubuntu, run sudo apt update, sudo apt upgrade and reboot the machine. Install NGINX with sudo apt install nginx, place the server certificate, key and CA chain somewhere, e.g. /home/ubuntu/certs/. In the file /etc/nginx/site-enabled/default comment everything out, we will not need it.
Next, create two files /etc/nginx/upstreamRest.conf and /etc/nginx/upstreamWeb.conf and add all FTS3 node ips to them. For /etc/nginx/upstreamRest.conf e.g.:

server 192.168.1.133:8446;
server 192.168.1.19:8446;

and for /etc/nginx/upstreamWeb.conf e.g.:

server 192.168.1.133:8449;
server 192.168.1.19:8449;

Next, replace the nginx.conf with an adjusted nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	#tcp_nopush on;
	#tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;
	resolver 8.8.8.8 8.8.4.4 valid=300s;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	#ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	#ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	upstream fts3web {
		include /etc/nginx/upstreamWeb.conf;
	}

	server {
		listen 80 default_server;
		listen [::]:80 default_server;
		server_name IP_OR_DNS_OF_SERVER;
		location / {
			return 301 https://$host$request_uri;
		}
	}

	server {
		listen 443 ssl http2;
		listen [::]:443 ssl http2;
		server_name IP_OR_DNS_OF_SERVER;
		ssl_certificate /PATH/TO/YOUR/server.cert.pem;
		ssl_certificate_key /PATH/TO/YOUR/server.key.pem;
		ssl_trusted_certificate /PATH/TO/YOUR/CA.pem;
		ssl_session_timeout 1d;
		ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
		ssl_session_tickets off;
		ssl_protocols TLSv1.3;
		ssl_prefer_server_ciphers off;
		add_header Strict-Transport-Security "max-age=63072000" always;

		location / {
			proxy_pass https://fts3web/;
			proxy_http_version 1.1;
			proxy_set_header Host $host:$server_port;
			proxy_set_header X-Forwarded-Host $server_name;
			proxy_set_header X-Real-IP $remote_addr;
			proxy_set_header X-Forwarded-Proto https;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "Upgrade";
			proxy_set_header Origin '';
		}
	}

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

stream {
        upstream fts3rest {
            include /etc/nginx/upstreamRest.conf;
        }

        server {
            listen 0.0.0.0:8446;
            ssl_certificate /PATH/TO/YOUR/server.cert.pem;
            ssl_certificate_key /PATH/TO/YOUR/server.key.pem;
            ssl_trusted_certificate /PATH/TO/YOUR/CA.pem;
            ssl_protocols TLSv1.2 TLSv1.3;
            ssl_prefer_server_ciphers off;
            proxy_pass fts3rest;
        }
}

Start NGINX with sudo systemctl start nginx and autostart with sudo systemctl enable nginx. Last, add a security group in OpenStack that enables ports 80, 443 for everyone and 8446 for the GitHub runner for more security or for everyone in case you want to access it from outside, e.g. for development.

You can test access by calling: curl --cert /PATH/TO/YOUR/client.cert.pem --key /PATH/TO/YOUR/client.key.pem --cacert /PATH/TO/YOUR/CA.pem https://PROXY_IP_OR_DNS:8446/whoami.
Output should be the DN of your client certificate and the roles.

Configure S3 Stratum 0 access

FTS3 required access to the S3 Stratum 0 bucket. With the client certificate, key and CA chain call:

curl --cert /PATH/TO/YOUR/client.cert.pem --key /PATH/TO/YOUR/client.key.pem --cacert /PATH/TO/YOUR/CA.pem https://PROXY_IP_OR_DNS:8446/config/cloud_storage -H "Content-Type: application/json" -X POST -d '{"storage_name":"S3:YOUR_S3_ACCESS_URL"}'
curl --cert /PATH/TO/YOUR/client.cert.pem --key /PATH/TO/YOUR/client.key.pem --cacert /PATH/TO/YOUR/CA.pem https://PROXY_IP_OR_DNS:8446/config/cloud_storage/S3:YOUR_S3_ACCESS_URL" -H "Content-Type: application/json" -X POST -d '{"vo_name":"","access_key":"S3_ACCESS_KEY","secret_key":"S3_SECRET_KEY","user_dn":"USER_DN_FROM_CLIENT_CERT","vo_role":""}'

YOUR_S3_ACCESS_URL should be without port, for example OPENSTACK.CEBITEC.UNI-BIELEFELD.DE. For the USER_DN see the database section.

Clone this wiki locally