Merge branch 'release/1.11'

README.md

@@ -5,13 +5,15 @@ Log issues and see general documentation at [dockstore](https://github.com/ga4gh
 Port 80 is exposed over http. This port should not be exposed to the public. A separately [configured load
 balancer](https://github.com/dockstore/dockstore-deploy) is responsible for SSL termination and forwarding traffic to this instance. Previously this repo handled the SSL termination with nginx and LetsEncrypt.
 
+If you are looking for how to run Dockstore locally as a developer, you are probably in the wrong place and should take a look at https://github.com/dockstore/dockstore/blob/develop/docker-compose.yml
+
 ## Prerequisities
 
-1. Tested on Ubuntu 16.04.3 LTS
-1. At least 20GB of disk space, 4GB of RAM, and two CPUs
+1. Tested on Ubuntu 20.04
+1. At least 20GB of disk space, 16GB of RAM, and 4 CPUs
 1. Docker setup following [https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/) including the post-installation steps for running without sudo
 1. The running Dockstore website will require ports 80 and 443 by default
-1. A client id and client secret for each of the integrations you wish to setup, github and quay.io as a minimum probably. You will need client ids and secrets for each integration as documented at the [Dockstore Primer](https://wiki.oicr.on.ca/display/SEQWARE/Dockstore+Primer#DockstorePrimer-SettingupDockstoreonyourcomputerfordevelopment(AssumingUbuntu)).
+1. A client id and client secret for each of the integrations you wish to setup, github and quay.io as a minimum probably. You will need client ids and secrets for each integration as documented on the internal [wiki](https://wiki.oicr.on.ca/display/DOC/OAuth+Apps+and+Other+3rd+Party+Registration).
 
 ## Usage
 
@@ -32,59 +34,14 @@ rebuild your docker images without affecting existing running containers
 
 4. After following the instructions in the bootstrap script and starting up the site with `docker-compose`, you can browse to the Dockstore site hosted at port 443 by default. `https://<domain-name>` if you specified https or `http://<domain-name>:443` if you did not. 
 
-5.  Note that the following volumes are created, `composesetup_esdata1` for ephermeral elastic search data, `composesetup_log_volume` for logging, and `composesetup_ui2_content` for storing the built UIs before they are handed off the nginx for service. 
+The current setup relies upon an externally hosted database (currently AWS RDS) and externally hosted search (currently AWS Elasticsearch). 
 
-6. For database backups, you can use a script setup in the cron for the host
-
-```
-@daily 		(echo '['`date`'] Nightly Back-up' && /home/ubuntu/compose_setup/scripts/postgres_backup.sh) 2>&1 | tee -a /home/ubuntu/compose_setup/scripts/ds_backup.log
-```
-
-This relies upon an IAM role for the appropriate S3 bucket. You will also need the AWS cli installed via ` sudo apt-get install awscli`. Note that this may not be readily apparent since a cron has a limited $PATH and it seems easy to accidentally get the awscli installed for specific users. 
-
 ### Loading Up a Database ###
 
-The docker-compose setup uses a mount from the host to keep the postgres database persistent (which is different from elastic search which is not) 
-
-However, this does require a convoluted way to add content to the DB as follows
-
-```
-docker-compose down
-# needed since dropping the schema can still leave some user information behind
-sudo rm -Rf postgres-data/
-nohup docker-compose up --force-recreate --remove-orphans &
-docker cp /tmp/backup.sql <container>:/tmp
-docker exec -ti <container> /bin/bash
-su - postgres
-psql
-DROP SCHEMA public CASCADE;
-CREATE SCHEMA public;
-CREATE USER dockstore WITH password 'dockstore';
-ALTER DATABASE postgres OWNER to dockstore;
-ALTER SCHEMA public OWNER to dockstore;
-\quit
-psql postgres -U dockstore -f  /tmp/backup.sql 
-# exit container (ctrl+d) and then run migration using newly loaded DB
-docker-compose down
-nohup docker-compose up --force-recreate --remove-orphans &
-```
-
+Loading up a database is usually not necessary since AWS RDS is persistent. Refer to https://github.com/dockstore/dockstore-deploy#database-setup 
 
 Note that database migration is run once during the startup process and is controlled via the `DATABASE_GENERATED` variable. Answer `yes` if you are working as a developer and want to start work from scratch from an empty database. Answer `no` if you are working as an administrator and/or wish to start Dockstore from a production or staging copy of the database.
 
-### Modifying the Database ###
-
-If direct modification of the database is required, e.g, a curator needs to modify the value of some row/column that is not accessible via the API, you can use the same steps as above, except for dropping the schema.
-
-This should be exercised with extreme caution, and with someone looking over your shoulder, as you have the potential to unintentionally overwrite or delete data. If you wish to proceed:
-
-```
-# Assuming you copied a file `fix.sql` to the /tmp directory:
-docker cp /tmp/fix.sql <container>:/tmp
-docker exec -ti <container> /bin/bash
-su - postgres
-psql -f  /tmp/fix.sql 
-```
 
 ## Logging Usage
 

docker-compose.yml

@@ -2,33 +2,8 @@ version: '2'
 
 services:
 
-  elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.3
-    environment:
-      - cluster.name=docker-cluster
-      - bootstrap.memory_lock=true
-      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
-    ulimits:
-      memlock:
-        soft: -1
-        hard: -1
-    mem_limit: 3g
-    restart: always
-    volumes:
-      - esdata1:/usr/share/elasticsearch/data
-      - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
-      - ./config/jvm.options.dockstore.es:/usr/share/elasticsearch/config/jvm.options
-    expose:
-      - "9200"
-      - "9300"
-    logging:
-      driver: "awslogs"
-      options:
-        awslogs-group: ${LOG_GROUP_NAME}
-        awslogs-stream: "elasticsearch"
-
   migration: 
-    image: quay.io/dockstore/dockstore-webservice:${DOCKSTORE_VERSION}
+    image: quay.io/dockstore/dockstore-webservice${DOCKSTORE_VERSION}
     volumes:
       - log_volume:/dockstore_logs
       - ./config/web.yml:/home/web.yml
@@ -41,10 +16,9 @@ services:
         awslogs-stream: "migration"
 
   webservice: 
-    image: quay.io/dockstore/dockstore-webservice:${DOCKSTORE_VERSION}
+    image: quay.io/dockstore/dockstore-webservice${DOCKSTORE_VERSION}
     restart: always
     depends_on:
-      - elasticsearch
       - migration
     volumes:
       - log_volume:/dockstore_logs

install_bootstrap

@@ -34,8 +34,14 @@ DOCKSTORE_DBPASSWORD=''
 DOCKSTORE_VERSION=''
 DOCUMENTATION_URL=''
 DOMAIN_NAME=''
+ELASTICSEARCH_DOMAIN=''
+ELASTICSEARCH_PASSWORD=''
+ELASTICSEARCH_PORT=''
+ELASTICSEARCH_PROTOCOL=''
+ELASTICSEARCH_USER=''
 EXTERNAL_GOOGLE_CLIENT_PREFIX1=''
 FEATURED_CONTENT_URL=''
+FEATURED_NEWS_URL=''
 GALAXY_PLUGIN_VERSION=''
 GITHUB_APP_ID=''
 GITHUB_APP_NAME=''
@@ -52,6 +58,7 @@ LOGSTASH_HOST=''
 ORCID_CLIENT_ID=''
 ORCID_CLIENT_SECRET=''
 ORCID_URL=''
+ORCID_SCOPE=''
 POSTGRES_DBPASSWORD=''
 PRODUCTION=''
 PUBLIC_LAUNCHER_IP_ADDRESS=''
@@ -247,7 +254,7 @@ done
 
 #Read the config file if it exists and then show previous values (but do not override variables in this script)
 if [ -f dockstore_launcher_config/compose.config ] ; then
-    source <(jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" dockstore_launcher_config/compose.config | grep -v "CONSONANCE")
+    source <(jq -r 'to_entries|map("\(.key)=\"\(.value|tostring)\"")|.[]' dockstore_launcher_config/compose.config | grep -v "CONSONANCE")
 fi
 
 while [[ "${run_dockstore_launcher^^}" != 'Y' &&  "${run_dockstore_launcher^^}" != 'N' && "$@" != *"--script"* ]] ; do
@@ -315,8 +322,26 @@ while [[ "${run_dockstore_launcher^^}" != 'Y' &&  "${run_dockstore_launcher^^}"
         discourse_category_id='discourse_category_id'
         ask_question "What is your Discourse category id (10 for staging, 11 for production)?" "$DISCOURSE_CATEGORY_ID" "discourse_category_id" $discourse_category_id
 
+        elasticsearch_domain='elasticsearch_domain'
+        ask_question "What is the domain name of the elasticsearch server (can be an localhost for development)?" "$ELASTICSEARCH_DOMAIN" "Elasticsearch domain name" $elasticsearch_domain
+
+        elasticsearch_protocol='elasticsearch_protocol'
+        ask_question "What is the protocol of the elasticsearch server (http or https)?" "$ELASTICSEARCH_PROTOCOL" "Elasticsearch protocol" $elasticsearch_protocol
+
+        elasticsearch_port='elasticsearch_port'
+        ask_question "What is the port of the elasticsearch server (9200 for local dev, 443 for https)?" "$ELASTICSEARCH_PORT" "Elasticsearch port" $elasticsearch_port
+
+        elasticsearch_user='elasticsearch_user'
+        ask_question "What is the master user of the elasticsearch server?" "$ELASTICSEARCH_USER" "Elasticsearch user" $elasticsearch_user
+
+        elasticsearch_password='elasticsearch_password'
+        ask_question "What is the master user's password on the elasticsearch server?" "$ELASTICSEARCH_PASSWORD" "Elasticsearch password" $elasticsearch_password
+
         featured_content_url='featured_content_url'
-        ask_question "What is your Featured Content URL (ex. https://s3.amazonaws.com/dockstore.featured.content/develop/feat-content.html for staging, https://s3.amazonaws.com/dockstore.featured.content/production/feat-content.html for production)?" "$FEATURED_CONTENT_URL" "featured_content_url" $featured_content_url
+        ask_question "What is your Featured Content URL (ex. https://content.dockstore.org/develop/feat-content.html for staging, https://content.dockstore.org/master/feat-content.html for production)?" "$FEATURED_CONTENT_URL" "featured_content_url" $featured_content_url
+
+        featured_news_url='featured_news_url'
+        ask_question "What is your Featured News URL (ex. https://content.dockstore.org/develop/news.html for staging, https://content.dockstore.org/master/news.html for production)?" "$FEATURED_CONTENT_URL" "featured_content_url" $featured_content_url
 
         compose_setup_version='compose_setup_version'
         ask_question "What is the compose setup version (commit id or tag)?" "$COMPOSE_SETUP_VERSION" "compose_setup_version" $compose_setup_version
@@ -337,6 +362,8 @@ while [[ "${run_dockstore_launcher^^}" != 'Y' &&  "${run_dockstore_launcher^^}"
         ask_question "What is your orcid client secret?" "$ORCID_CLIENT_SECRET" "orcid_client_secret" $orcid_client_secret
         orcid_url='orcid_url'
 	    ask_question "What is the orcid url (https://sandbox.orcid.org for staging, https://orcid.org for production)?" "$ORCID_URL" "orcid_url" $orcid_url
+        orcid_scope='orcid_scope'
+        ask_question "What is your orcid scope?" "$ORCID_SCOPE" "orcid_scope" $orcid_scope
 
         tooltester_bucket_name='tooltester_bucket_name'
         ask_question "What is the name of the bucket holding tooltester logs?" "$TOOLTESTER_BUCKET_NAME" "tooltester_bucket_name" $tooltester_bucket_name
@@ -403,6 +430,11 @@ while [[ "${run_dockstore_launcher^^}" != 'Y' &&  "${run_dockstore_launcher^^}"
 "BITBUCKET_CLIENT_SECRET":"${bitbucket_client_secret}",
 "COMPOSE_SETUP_VERSION":"${compose_setup_version}",
 "DOMAIN_NAME":"${domain_name}",
+"ELASTICSEARCH_DOMAIN"="${elasticsearch_domain}",
+"ELASTICSEARCH_PASSWORD"="${elasticsearch_password}",
+"ELASTICSEARCH_PORT"="${elasticsearch_port}",
+"ELASTICSEARCH_PROTOCOL"="${elasticsearch_protocol}",
+"ELASTICSEARCH_USER"="${elasticsearch_user}",
 "HTTPS":${https},
 "DEPLOY_VERSION":"${deploy_version}",
 "DISCOURSE_URL":"${discourse_url}",
@@ -431,6 +463,7 @@ while [[ "${run_dockstore_launcher^^}" != 'Y' &&  "${run_dockstore_launcher^^}"
 "ORCID_CLIENT_ID":"${orcid_client_id}",
 "ORCID_CLIENT_SECRET":"${orcid_client_secret}",
 "ORCID_URL":"${orcid_url}",
+"ORCID_SCOPE": "${orcid_scope}",
 "ZENODO_CLIENT_ID":"${zenodo_client_id}",
 "ZENODO_CLIENT_SECRET":"${zenodo_client_secret}",
 "ZENODO_URL":"${zenodo_url}",

templates/default.nginx_http.security.conf.template

@@ -2,7 +2,7 @@
 server_tokens off;
 
 # Don't allow page to be rendered inside a frame or iframe
-add_header X-Frame-Options DENY;
+add_header X-Frame-Options DENY always;
 
 # Certificate Transparency is a way of cross-referencing certificates with a log of all certificates issued,
 # to make sure they are authentic. This will become obsolete in June 2021.
@@ -15,15 +15,15 @@ add_header X-XSS-Protection "1; mode=block" always;
 
 # Enable a referrer policy that protects users' privacy while still enabling
 # Dockstore to see how users interact with the site.
-add_header Referrer-Policy "same-origin";
+add_header Referrer-Policy "strict-origin-when-cross-origin" always;
 
 # Explicitly list domains allowed to serve content for this site
-#add_header Content-Security-Policy-Report-Only "report-uri https://api.dockstore-security.org/csp-report; default-src 'self'; object-src 'none'; base-uri 'self'; manifest-src 'self'; media-src 'self'; worker-src 'none'; script-src 'report-sample' 'self' 'unsafe-hashes' 'unsafe-inline' discuss.dockstore.org gui.dockstore.org platform.twitter.com www.google-analytics.com www.googletagmanager.com; style-src 'report-sample' 'self' 'unsafe-inline' cdnjs.cloudflare.com fonts.googleapis.com gui.dockstore.org; connect-src 'self' s3.amazonaws.com api.github.com view.commonwl.org; font-src 'self' fonts.gstatic.com gui.dockstore.org; frame-src 'self' discuss.dockstore.org; img-src 'self' avatars0.githubusercontent.com avatars1.githubusercontent.com avatars3.githubusercontent.com camo.githubusercontent.com gui.dockstore.org i.imgur.com img.shields.io quay.io via.placeholder.com www.googletagmanager.com www.gravatar.com;";
+add_header Content-Security-Policy-Report-Only "report-uri https://api.dockstore-security.org/csp-report; default-src 'self'; object-src 'none'; base-uri 'self'; manifest-src 'self'; media-src 'self'; worker-src 'none'; script-src 'report-sample' 'self' 'unsafe-hashes' 'unsafe-inline' 'unsafe-eval' discuss.dockstore.org gui.dockstore.org *.twitter.com *.twimg.com www.google-analytics.com www.googletagmanager.com; style-src 'report-sample' 'self' 'unsafe-inline' cdnjs.cloudflare.com fonts.googleapis.com *.twitter.com *.twimg.com gui.dockstore.org; connect-src 'self' s3.amazonaws.com api.github.com view.commonwl.org www.google-analytics.com gui.dockstore.org; font-src 'self' fonts.gstatic.com gui.dockstore.org; frame-src 'self' discuss.dockstore.org platform.twitter.com; img-src data: 'self' avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com camo.githubusercontent.com gui.dockstore.org i.imgur.com api.travis-ci.com img.shields.io quay.io via.placeholder.com *.wp.com *.googleusercontent.com www.googletagmanager.com www.google-analytics.com www.gravatar.com *.twitter.com *.twimg.com;" always;
 
 # Hide server header
 proxy_hide_header Server;
 
 # Protect against MIME sniffing
-add_header X-Content-Type-Options "nosniff";
+add_header X-Content-Type-Options "nosniff" always;
 
-add_header Strict-Transport-Security $hsts_header;
+add_header Strict-Transport-Security $hsts_header always;

templates/elasticsearch.yml

@@ -11,4 +11,4 @@ script.allowed_contexts: none
 rest.action.multi.allow_explicit_index: true
 # default off, not idea, but matches elastic search 5.
 xpack.security.enabled: false
-xpack.monitoring.enabled: false 
+xpack.watcher.enabled: false

templates/init_migration.sh.template

@@ -4,12 +4,12 @@
 cd "$(dirname "$0")"
 
 {{#DATABASE_GENERATED}}
-java -Ddw.database.user=postgres -Ddw.database.password={{ POSTGRES_DBPASSWORD }} -jar dockstore-webservice-*.jar db migrate web.yml --include 1.3.0.generated,1.3.1.consistency,1.4.0,1.5.0,1.6.0,1.7.0 | tee --append /dockstore_logs/webservice.out
+java -Ddw.database.user=postgres -Ddw.database.password="{{{ POSTGRES_DBPASSWORD }}}" -jar dockstore-webservice-*.jar db migrate web.yml --include 1.3.0.generated,1.3.1.consistency,1.4.0,1.5.0,1.6.0,1.7.0 | tee --append /dockstore_logs/webservice.out
 {{/DATABASE_GENERATED}}
 {{^DATABASE_GENERATED}}
-java -Ddw.database.user=postgres -Ddw.database.password={{ POSTGRES_DBPASSWORD }} -jar dockstore-webservice-*.jar db migrate web.yml --include 1.3.1.consistency,1.4.0,1.5.0,1.6.0,1.7.0 | tee --append /dockstore_logs/webservice.out
+java -Ddw.database.user=postgres -Ddw.database.password="{{{ POSTGRES_DBPASSWORD }}}" -jar dockstore-webservice-*.jar db migrate web.yml --include 1.3.1.consistency,1.4.0,1.5.0,1.6.0,1.7.0 | tee --append /dockstore_logs/webservice.out
 {{/DATABASE_GENERATED}}
 # this particular migration needs to run as postgres because only postgres can surrender ownership
-java -Ddw.database.user=postgres -Ddw.database.password={{ POSTGRES_DBPASSWORD }} -jar dockstore-webservice-*.jar db migrate web.yml --include 1.7.0.relinquish
+java -Ddw.database.user=postgres -Ddw.database.password="{{{ POSTGRES_DBPASSWORD }}}" -jar dockstore-webservice-*.jar db migrate web.yml --include 1.7.0.relinquish
 # future migrations will start here and should be run as dockstore
-java -Ddw.database.user=dockstore -Ddw.database.password="{{ DOCKSTORE_DBPASSWORD }}" -jar dockstore-webservice-*.jar db migrate web.yml --include 1.8.0,1.9.0,1.10.0 | tee --append /dockstore_logs/webservice.out
+java -Ddw.database.user=dockstore -Ddw.database.password="{{{ DOCKSTORE_DBPASSWORD }}}" -jar dockstore-webservice-*.jar db migrate web.yml --include 1.8.0,1.9.0,1.10.0,1.11.0 | tee --append /dockstore_logs/webservice.out

templates/jvm.options.dockstore.es.template

@@ -33,9 +33,9 @@
 ################################################################
 
 ## GC configuration
--XX:+UseConcMarkSweepGC
--XX:CMSInitiatingOccupancyFraction=75
--XX:+UseCMSInitiatingOccupancyOnly
+8-13:-XX:+UseConcMarkSweepGC
+8-13:-XX:CMSInitiatingOccupancyFraction=75
+8-13:-XX:+UseCMSInitiatingOccupancyOnly
 
 ## optimizations
 

templates/web.yml.template

@@ -39,8 +39,11 @@ limitConfig:
   workflowVersionLimit: 50
 
 esconfiguration:
-  port: 9200
-  hostname: elasticsearch
+  port: {{ ELASTICSEARCH_PORT }}
+  hostname: {{ ELASTICSEARCH_DOMAIN }}
+  protocol: {{ ELASTICSEARCH_PROTOCOL }}
+  user: {{ ELASTICSEARCH_USER }}
+  password: {{{ ELASTICSEARCH_PASSWORD }}}
 
 authorizerType: {{ AUTHORIZER_TYPE }}
 externalGoogleClientIdPrefixes:
@@ -96,15 +99,15 @@ database:
   user: dockstore
 
   # the password
-  password: {{ DOCKSTORE_DBPASSWORD }}
+  password: {{{ DOCKSTORE_DBPASSWORD }}}
 
   # the JDBC URL
   url: jdbc:postgresql://{{ DATABASE_DOMAIN }}:5432/postgres
 
   # any properties specific to your JDBC driver:
   properties:
     charSet: UTF-8
-    hibernate.dialect: org.hibernate.dialect.PostgreSQLDialect
+    hibernate.dialect: org.hibernate.dialect.PostgreSQL10Dialect
     # create database as needed, disable in production
     hibernate.hbm2ddl.auto: validate
 
@@ -177,10 +180,11 @@ uiConfig:
 
   orcidAuthUrl: {{ ORCID_URL }}/oauth/authorize
   orcidRedirectPath: /auth/orcid.org
-  orcidScope: /authenticate
+  orcidScope: {{ ORCID_SCOPE }}
 
   documentationUrl: {{ DOCUMENTATION_URL }}
   featuredContentUrl: {{ FEATURED_CONTENT_URL }}
+  featuredNewsUrl: {{ FEATURED_NEWS_URL }}
 
-  composeSetupVersion: {{ COMPOSE_SETUP_VERSION }}
-  deployVersion: {{ DEPLOY_VERSION }}
+  composeSetupVersion: "{{ COMPOSE_SETUP_VERSION }}"
+  deployVersion: "{{ DEPLOY_VERSION }}"