Add extra exercise to include host OS logging

README.rst

@@ -438,6 +438,33 @@ but they are not here).
 Once you're in, Kibana needs some further setup which is not automated.
 Set the log index to ``flog-*`` and you should be ready to go.
 
+Extra: Adding Host OS Logs
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We can extend the catchment of centralised logging to include logs
+generated by the host OS and processed using ``rsyslog``.  It is
+relatively straightforward to connect ``rsyslog`` to ``fluentd``:
+Kolla-Ansible will configure ``fluentd`` to bind to a UDP port and
+listen on each host's primary network interface.  It is easy to
+configure ``rsyslog`` to forward log messages to this UDP port.
+
+The best approach is to use a `Kayobe custom playbook
+<https://docs.openstack.org/kayobe/latest/custom-ansible-playbooks.html>`_
+to make this change.  Custom playbooks have access to the Kayobe inventory
+and group variables.  We can use the ``overcloud`` group to reconfigure
+``rsyslog`` on all hosts.
+
+To run the custom playbook:
+
+.. code-block:: console
+
+    cd config/src/kayobe-confg/etc/kayobe/ansible
+    kayobe playbook run host-logging.yml
+
+Once the playbook completes, you should soon find extra log messages
+in Kibana from host OS services such as ``kernel``, ``systemd`` and
+``sshd``.
+
 Adding the Barbican service
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

etc/kayobe/ansible/host-logging.yml

@@ -0,0 +1,27 @@
+---
+# Direct host OS logs to Fluentd for forwarding to ElasticSearch
+- name: Configure rsyslog to forward messages
+  hosts: overcloud
+  become: yes
+
+  tasks:
+  - name: Check rsyslog is started and enabled
+    systemd:
+      state: started
+      enabled: yes
+      name: rsyslog
+
+  - name: Update rsyslog file
+    lineinfile:
+      path: /etc/rsyslog.conf
+      insertafter: '^#*.* @@remote-host:514'
+      line: '*.* @{{ ansible_default_ipv4.address }}:5140'
+    register: rsyslog_config
+
+  - name: Restart rsyslog
+    systemd:
+      state: restarted
+      name: rsyslog
+    when: rsyslog_config.changed
+
+...