Cannot access private Matrix server (certificate error)

[cerebrate]

I have an internal Matrix server which I'd like to use for Octoslack notifications, but I can't connect to the server because it uses a certificate issued by a local (internal) CA rather than a default CA:

2021-01-10 15:37:44,297 - octoprint.plugins.tracking - INFO - Sent tracking event print_started, payload: {'origin': 'local', 'file': 'ca2357b0723f9cf4d60796f8bfac2d507876ef10'}
2021-01-10 15:37:44,460 - octoprint.plugins.Octoslack - ERROR - Matrix send error: Something went wrong in GET requesting https://matrix.harmony.arkane-systems.lan//_matrix/client/r0/sync: HTTPSConnectionPool(host='matrix.harmony.arkane-systems.lan', port=443): Max retries exceeded with url: //_matrix/client/r0/sync?timeout=30000&filter=%7B+%22room%22%3A+%7B+%22timeline%22+%3A+%7B+%22limit%22+%3A+20+%7D+%7D+%7D&access_token=MDAyMGxvY2F0aW9uIGFya2FuZS1zeXN0ZW1zLm5ldAowMDEzaWRlbnRpZmllciBrZXkKMDAxMGNpZCBnZW4gPSAxCjAwMzBjaWQgdXNlcl9pZCA9IEBvY3RvcHJpbnQ6YXJrYW5lLXN5c3RlbXMubmV0CjAwMTZjaWQgdHlwZSA9IGFjY2VzcwowMDIxY2lkIG5vbmNlID0gfm5OfkNeTThLbzE3MV9PWgowMDJmc2lnbmF0dXJlIOxjCQ1iLcYxYDP4sL8SNc6NbFr7EEoi2Tt7XD3IEOtJCg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/pi/oprint/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='matrix.harmony.arkane-systems.lan', port=443): Max retries exceeded with url: //_matrix/client/r0/sync?timeout=30000&filter=%7B+%22room%22%3A+%7B+%22timeline%22+%3A+%7B+%22limit%22+%3A+20+%7D+%7D+%7D&access_token=MDAyMGxvY2F0aW9uIGFya2FuZS1zeXN0ZW1zLm5ldAowMDEzaWRlbnRpZmllciBrZXkKMDAxMGNpZCBnZW4gPSAxCjAwMzBjaWQgdXNlcl9pZCA9IEBvY3RvcHJpbnQ6YXJrYW5lLXN5c3RlbXMubmV0CjAwMTZjaWQgdHlwZSA9IGFjY2VzcwowMDIxY2lkIG5vbmNlID0gfm5OfkNeTThLbzE3MV9PWgowMDJmc2lnbmF0dXJlIOxjCQ1iLcYxYDP4sL8SNc6NbFr7EEoi2Tt7XD3IEOtJCg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/matrix_client/api.py", line 670, in _send
    verify=self.validate_cert
  File "/home/pi/oprint/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/pi/oprint/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/pi/oprint/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='matrix.harmony.arkane-systems.lan', port=443): Max retries exceeded with url: //_matrix/client/r0/sync?timeout=30000&filter=%7B+%22room%22%3A+%7B+%22timeline%22+%3A+%7B+%22limit%22+%3A+20+%7D+%7D+%7D&access_token=MDAyMGxvY2F0aW9uIGFya2FuZS1zeXN0ZW1zLm5ldAowMDEzaWRlbnRpZmllciBrZXkKMDAxMGNpZCBnZW4gPSAxCjAwMzBjaWQgdXNlcl9pZCA9IEBvY3RvcHJpbnQ6YXJrYW5lLXN5c3RlbXMubmV0CjAwMTZjaWQgdHlwZSA9IGFjY2VzcwowMDIxY2lkIG5vbmNlID0gfm5OfkNeTThLbzE3MV9PWgowMDJmc2lnbmF0dXJlIOxjCQ1iLcYxYDP4sL8SNc6NbFr7EEoi2Tt7XD3IEOtJCg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/oprint/lib/python3.7/site-packages/octoprint_Octoslack/__init__.py", line 3851, in send_slack_message
    user_id=matrixUserID,
  File "/home/pi/oprint/lib/python3.7/site-packages/matrix_client/client.py", line 152, in __init__
    self._sync()
  File "/home/pi/oprint/lib/python3.7/site-packages/matrix_client/client.py", line 555, in _sync
    response = self.api.sync(self.sync_token, timeout_ms, filter=self.sync_filter)
  File "/home/pi/oprint/lib/python3.7/site-packages/matrix_client/api.py", line 105, in sync
    api_path=MATRIX_V2_API_PATH)
  File "/home/pi/oprint/lib/python3.7/site-packages/matrix_client/api.py", line 673, in _send
    raise MatrixHttpLibError(e, method, endpoint)
matrix_client.errors.MatrixHttpLibError: Something went wrong in GET requesting https://matrix.harmony.arkane-systems.lan//_matrix/client/r0/sync: HTTPSConnectionPool(host='matrix.harmony.arkane-systems.lan', port=443): Max retries exceeded with url: //_matrix/client/r0/sync?timeout=30000&filter=%7B+%22room%22%3A+%7B+%22timeline%22+%3A+%7B+%22limit%22+%3A+20+%7D+%7D+%7D&access_token=MDAyMGxvY2F0aW9uIGFya2FuZS1zeXN0ZW1zLm5ldAowMDEzaWRlbnRpZmllciBrZXkKMDAxMGNpZCBnZW4gPSAxCjAwMzBjaWQgdXNlcl9pZCA9IEBvY3RvcHJpbnQ6YXJrYW5lLXN5c3RlbXMubmV0CjAwMTZjaWQgdHlwZSA9IGFjY2VzcwowMDIxY2lkIG5vbmNlID0gfm5OfkNeTThLbzE3MV9PWgowMDJmc2lnbmF0dXJlIOxjCQ1iLcYxYDP4sL8SNc6NbFr7EEoi2Tt7XD3IEOtJCg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

I've added the local CA root certificate to the Octoprint host and run update-ca-certificates, and running a few manual tests in python with the requests library suggests that python can validate the certificate correctly, at least for a simple requests.get().

How can I get this CA cert into/used by Octoslack, or failing that as a workaround, disable SSL verification temporarily?

[cerebrate]

I can work around this temporarily by patching oprint/lib/python3.7/site-packages/matrix_client/api.py to include the last line below:

    def __init__(self, base_url, token=None, identity=None, default_429_wait_ms=5000):
        self.base_url = base_url
        self.token = token
        self.identity = identity
        self.txn_id = 0
        self.validate_cert = True
        self.session = Session()
        self.default_429_wait_ms = default_429_wait_ms
        self.session.verify = '/etc/ssl/certs/ca-certificates.crt'

but that's hardly a fix. Since the problem is with matrix_client and not with Octoslack, I should report this over there, but since on the other hand matrix_client is almost-but-not-quite deprecated in favor of matrix-nio and they're not taking bug fixes or feature requests at this time, I'm leaving it open here.