Skip to content

Commit

Permalink
migrate discourse to EL9
Browse files Browse the repository at this point in the history
* discourse el9 setup
* update documentation of discourse
  • Loading branch information
evgeni authored Sep 27, 2023
1 parent a1abebb commit 4bf230e
Show file tree
Hide file tree
Showing 10 changed files with 368 additions and 19 deletions.
11 changes: 11 additions & 0 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,15 @@ Vagrant.configure("2") do |config|
provider.vm.box_url = CENTOS_9_BOX_URL
end
end

config.vm.define "discourse" do |override|
override.vm.hostname = "discourse"
override.vm.box = "centos/stream9"

override.vm.provider "libvirt" do |libvirt, provider|
libvirt.memory = "2048"
libvirt.machine_virtual_size = 40
provider.vm.box_url = CENTOS_9_BOX_URL
end
end
end
43 changes: 28 additions & 15 deletions docs/discourse.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,21 @@

## Host - community.theforeman.org

Discourse runs on our Scaleway account, and is linked to our Puppet infra so the usual people have SSH key access. Backups to the puppetserver of the DB dumps, attachments, and container definitions are enabled too.
Discourse runs in a VM on our Conova hypvervisor, and is linked to our Puppet infra so the usual people have SSH key access.
Backups of the DB dumps and attachments are done to our normal backup infrastructure daily.

## What's running?

Discourse runs in Docker, so very little is running at the main host level. There are 2 containers running:

1. The main Discourse app, containing Rails, Redis, Sidekiq, and PSQL. Listens on 80 and 443 and provides it's own Let's Encrypt setup
1. The main Discourse app, containing Rails, Redis, Sidekiq, and PostgreSQL. Listens on 80 and 443 and provides it's own Let's Encrypt setup
2. A Postfix container listening on 25 which handles incoming mail and then passes it to Discourse for turning into posts

## Locations

All the work is done in `/var/discourse`. In that directory you'll find:

* `containers` directory which contains YAML files specifying the `app` and `mail-receiver` definitions
* `containers` directory which contains YAML files specifying the `app` and `mail-receiver` definitions (managed by the `discourse` Puppet module)
* `shared` directory containing the permanent volumes mounted into Docker (backups are in here)
* `launcher` bash script for interacting with the containers

Expand All @@ -25,7 +26,7 @@ All the work is done in `/var/discourse`. In that directory you'll find:

There's really only one useful command here - `./launcher logs` will get you the recent logs from the container. Combined with grep, you can check to see if a specific mail arrived, eg:

```
```console
$ ./launcher logs mail-receiver | egrep "from=|receive-mail"
<22>Dec 26 06:19:44 postfix/qmgr[80]: 54787280723: from=<email-redacted>, size=10324, nrcpt=1 (queue active)
<23>Dec 26 06:19:44 receive-mail[17867]: Recipient: reply+<token-redacted>`community.theforeman.org
Expand All @@ -39,23 +40,35 @@ For basic app control, `start`, `stop`, and `restart` are available, of course.

#### restarting

cd /var/discourse ; sudo ./launcher restart app

```console
$ cd /var/discourse ; sudo ./launcher restart app
```

If for some reason the Discourse WebUI upgrade process fails, it will direct you to do a CLI upgrade, which is usually `./launcher rebuild app` but on-screen notes are usually provided.

## Disaster recovery

There are two forms of backup:
## Backup and Restore

### Volume snapshots
### Backup

Every day [a script](https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/scaleway/files/manage_snapshots.rb) runs on the Discourse VM which talks to the Scaleway Snapshot API, triggers a new snapshot, and then (on success) cleans the older images away. The most recent 3 snapshots are retained.
Discourse itself performs a backup daily (see Admin > Backups).
These backups are then transferred to our normal backup infrastructure.
They contain all the content for Discourse itself (database, files).

In the event of the VM completely dying, this is your starting point, as it should be a complete image of the whole system, with all API keys etc.
### Restore

### Discourse content tarballs
Backups can be restored by setting up a new Discourse instance, copying the tarball to the backups folder, and then restoring from within the UI or console.
Note you have to enable Restore from the Settings UI or console.

Discourse itself performs a backup daily (see Admin > Backups), which runs ~12 hours after the snapshot above, to provide staggered cover. This contains all the content for Discourse itself, and can be restored by setting up a new Discourse instance, copying the tarball to the backups folder, and then restoring from within the UI. Note you have to enable Restore from the Settings UI.
#### Restore via console

The backup tarballs, along with the container definitions in `/var/discourse/containers` are backed up nightly to the Puppetmaster, in case of complete failure of the Scaleway system.
1. Create a fresh machine with the `discourse` Puppet module applied to it.
Make sure `community.theforeman.org` already points to that system or override `$hostname` in Puppet to make Let's Encrypt and friends work correctly.
2. Let Discourse rebuild the main app and start it: `./launcher rebuild app`.
When this is done, you will have an empty, fresh Discourse installation.
3. Obtain a backup tarball and place it into `/var/discourse/shared/standalone/backups/default`.
4. Enter the `app` container: `./launcher enter app`.
5. Enable restores: `discourse enable_restore`
6. Restore the backup: `discourse restore sitename-2019-02-03-042252-v20190130013015.tar.gz`
7. Leave the `app` container: `exit`
8. Rebuild the `app` to migrate the DB etc: `./launcher rebuild app`
9. Rebuild the `mail-receiver` container: `./launcher rebuild mail-receiver`
7 changes: 7 additions & 0 deletions puppet/data/vagrant.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
---
discourse::developer_emails: '[email protected]'
discourse::api_key: '1234567890abcdef'
discourse::le_account_email: '[email protected]'
discourse::smtp_address: 'mail.example.com'
discourse::smtp_user_name: 'discourse'
discourse::smtp_password: 'changeme'

restic::password: "SomethingVerySecret"

profiles::backup::receiver:
Expand Down
60 changes: 60 additions & 0 deletions puppet/modules/discourse/manifests/init.pp
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
class discourse(
String $developer_emails,
String $api_key,
String $le_account_email,
Stdlib::Host $smtp_address,
String $smtp_user_name,
String $smtp_password,
Stdlib::Port $smtp_port = 587,
Stdlib::Absolutepath $root = '/var/discourse',
Stdlib::Host $hostname = 'community.theforeman.org',
) {
ensure_packages(['git'])

vcsrepo { $root:
ensure => present,
provider => git,
source => 'https://github.com/discourse/discourse_docker.git',
}

$containers = "${root}/containers"

file { $containers:
ensure => directory,
owner => 'root',
group => 'root',
mode => '0700',
require => Vcsrepo[$root],
}

$app_context = {
'root' => $root,
'hostname' => $hostname,
'developer_emails' => $developer_emails,
'smtp_address' => $smtp_address,
'smtp_port' => $smtp_port,
'smtp_user_name' => $smtp_user_name,
'smtp_password' => $smtp_password,
'le_account_email' => $le_account_email,
}

file { "${containers}/app.yml":
owner => 'root',
group => 'root',
mode => '0700',
content => epp('discourse/app.yml.epp', $app_context),
}

$mail_context = {
'root' => $root,
'hostname' => $hostname,
'api_key' => $api_key,
}

file { "${containers}/mail-receiver.yml":
owner => 'root',
group => 'root',
mode => '0700',
content => epp('discourse/mail-receiver.yml.epp', $mail_context ),
}
}
111 changes: 111 additions & 0 deletions puppet/modules/discourse/templates/app.yml.epp
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
<%- |
Stdlib::Absolutepath $root,
Stdlib::Host $hostname,
String $developer_emails,
Stdlib::Host $smtp_address,
Stdlib::Port $smtp_port,
String $smtp_user_name,
String $smtp_password,
String $le_account_email,
| -%>
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
- "templates/web.ssl.template.yml"
- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
- "80:80" # http
- "443:443" # https

params:
db_default_text_search_config: "pg_catalog.english"

## Set db_shared_buffers to a max of 25% of the total memory.
## will be set automatically by bootstrap based on detected RAM, or you can override
db_shared_buffers: "256MB"

## can improve sorting performance, but adds memory usage per-connection
#db_work_mem: "40MB"

## Which Git revision should this container use? (default: tests-passed)
version: stable

env:
LANG: en_US.UTF-8
# DISCOURSE_DEFAULT_LOCALE: en

## How many concurrent web requests are supported? Depends on memory and CPU cores.
## will be set automatically by bootstrap based on detected CPUs, or you can override
UNICORN_WORKERS: 4

## TODO: The domain name this Discourse instance will respond to
DISCOURSE_HOSTNAME: <%= $hostname %>

## Uncomment if you want the container to be started with the same
## hostname (-h option) as specified above (default "$hostname-$config")
#DOCKER_USE_HOSTNAME: true

## TODO: List of comma delimited emails that will be made admin and developer
## on initial signup example '[email protected],[email protected]'
DISCOURSE_DEVELOPER_EMAILS: "<%= $developer_emails %>"

## TODO: The SMTP mail server used to validate new accounts and send notifications
DISCOURSE_SMTP_ADDRESS: <%= $smtp_address %>
DISCOURSE_SMTP_PORT: <%= $smtp_port %>
DISCOURSE_SMTP_USER_NAME: "<%= $smtp_user_name %>"
DISCOURSE_SMTP_PASSWORD: "<%= $smtp_password %>"
#DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)

## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
LETSENCRYPT_ACCOUNT_EMAIL: <%= $le_account_email %>

## The CDN address for this Discourse instance (configured to pull)
## see https://meta.discourse.org/t/14857 for details
#DISCOURSE_CDN_URL: //discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: <%= $root %>/shared/standalone
guest: /shared
- volume:
host: <%= $root %>/shared/standalone/log/var-log
guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-data-explorer.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/paviliondev/discourse-events.git
- git clone https://github.com/paviliondev/discourse-locations.git
- bash -c 'git clone https://github.com/discourse/discourse-checklist.git && cd discourse-checklist && git checkout 4a7f3df360a8e4ff3bbebfed33ea545b1c72506e'

## Any custom commands to run after building
run:
- exec: echo "Beginning of custom commands"
## If you want to set the 'From' email address for your first registration, uncomment and change:
## After getting the first signup email, re-comment the line. It only needs to run once.
#- exec: rails r "SiteSetting.notification_email='[email protected]'"
- exec: echo "End of custom commands"
45 changes: 45 additions & 0 deletions puppet/modules/discourse/templates/mail-receiver.yml.epp
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<%- |
Stdlib::Absolutepath $root,
Stdlib::Host $hostname,
String $api_key,
| -%>
## this is the incoming mail receiver container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild mail-receiver
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

base_image: discourse/mail-receiver:release
update_pups: false

expose:
- "25:25" # SMTP

env:
LANG: en_US.UTF-8

## Where e-mail to your forum should be sent. In general, it's perfectly fine
## to use the same domain as the forum itself here.
MAIL_DOMAIN: <%= $hostname %>

## The base URL for this Discourse instance.
## This will be whatever your Discourse site URL is. For example,
## https://discourse.example.com. If you're running a subfolder setup,
## be sure to account for that (ie https://example.com/forum).
DISCOURSE_BASE_URL: "https://<%= $hostname %>"

## The master API key of your Discourse forum. You can get this from
## the "API" tab of your admin panel.
DISCOURSE_API_KEY: "<%= $api_key %>"

## The username to use for processing incoming e-mail. Unless you have
## renamed the `system` user, you should leave this as-is.
DISCOURSE_API_USERNAME: system

volumes:
- volume:
host: <%= $root %>/shared/mail-receiver/postfix-spool
guest: /var/spool/postfix
26 changes: 22 additions & 4 deletions puppet/modules/profiles/manifests/discourse.pp
Original file line number Diff line number Diff line change
@@ -1,14 +1,32 @@
# @summary Manage a Discourse server
# @see https://github.com/discourse/discourse/blob/main/docs/INSTALL-cloud.md
class profiles::discourse {
$root = '/var/discourse'
# TODO: manage docker
# TODO: vcsrepo https://github.com/discourse/discourse_docker.git on $root
if $facts['os']['family'] == 'RedHat' {
yumrepo { 'docker-ce-stable':
descr => 'Docker CE Stable - $basearch',
baseurl => 'https://download.docker.com/linux/centos/$releasever/$basearch/stable',
gpgkey => 'https://download.docker.com/linux/centos/gpg',
}

ensure_packages(['docker-ce'], { require => Yumrepo['docker-ce-stable'] })

service { 'docker':
ensure => 'running',
enable => true,
require => Package['docker-ce'],
}

include discourse
$backup_path = ["${discourse::root}/shared/standalone/backups"]
} else {
$root = '/var/discourse'
$backup_path = ["${root}/containers", "${root}/shared/standalone/backups"]
}

include profiles::backup::sender

restic::repository { 'discourse':
backup_cap_dac_read_search => true,
backup_path => ["${root}/containers", "${root}/shared/standalone/backups"],
backup_path => $backup_path,
}
}
23 changes: 23 additions & 0 deletions puppet/spec/classes/discourse_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
require 'spec_helper'

describe 'discourse' do
on_supported_os.each do |os, facts|
context "on #{os}" do
let(:facts) { facts }
let(:pre_condition) do
<<~PUPPET
class { 'discourse':
developer_emails => '[email protected]',
api_key => '1234567890abcdef',
le_account_email => '[email protected]',
smtp_address => 'mail.example.com',
smtp_user_name => 'discourse',
smtp_password => 'changeme',
}
PUPPET
end

it { is_expected.to compile.with_all_deps }
end
end
end
Loading

0 comments on commit 4bf230e

Please sign in to comment.