Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Matrix Well Known URI Server Discovery #1198

Merged
merged 4 commits into from
Sep 16, 2024
Merged

Matrix Well Known URI Server Discovery #1198

merged 4 commits into from
Sep 16, 2024

Conversation

caronc
Copy link
Owner

@caronc caronc commented Sep 1, 2024 •
edited
Loading

Description:

Related issue (if applicable): #1067

Matrix Server Delegation Added.

If you set ?delegate=yes on the Apprise matrix:// URL, A DNS SRV query is made to _matrix._tcp.your-host to acquire a Delegated server. If one is found, then m.server parameters are added into all matrix subsequent calls.

@bwachtendorf, I would really love your feedback on this one 🙏

Checklist

  • The code change is tested and works locally.
  • There is no commented out code in this PR.
  • No lint errors (use flake8)
  • 100% test coverage

Testing

Anyone can help test this source code as follows:

# Create a virtual environment to work in as follows:
python3 -m venv apprise

# Change into our new directory
cd apprise

# Activate our virtual environment
source bin/activate

# Install the branch
pip install git+https://github.com/caronc/apprise.git@1067-matrix-delegation

# Test out the changes with the following command:
apprise -t "Test Title" -b "Test Message" \
  "matrix://user:pass@hostname/?delegate=yes"

@caronc caronc linked an issue Sep 1, 2024 that may be closed by this pull request
Matrix delegation not working #1067
Closed
@codecov Codecov
Copy link

codecov bot commented Sep 1, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.35%. Comparing base (fa6d4e6) to head (43a9bb5).
Report is 6 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #1198   +/-   ##
=======================================
  Coverage   99.35%   99.35%           
=======================================
  Files         147      147           
  Lines       20320    20414   +94     
  Branches     3979     3995   +16     
=======================================
+ Hits        20189    20283   +94     
  Misses        121      121           
  Partials       10       10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bwachtendorf
Copy link

Thanks for tackling this issue.

This will not work because you mixed up the SRV delegation with the well-known delegation.
I should have been more precise in my initial bug report.

My setup is using the well-known delegation and serves a JSON like the example from #1067.
Your fix tries to read that JSON from an SRV record which works differently and should look more like this: 10 0 8448 matrix.example.com. .

Additionally _matrix._tcp.your-host seems to be deprecated in favor of a slightly different SRV record.

See the matrix spec for details on how to get the matrix server hostname.

Finally everything I use that connects with matrix, seems to use that spec and figure out the actual server to connect to without any need to specify delegation and I think therefore ?delegate=yes should be the default and you should only be able to turn it off (if thats needed at all).

@caronc
Copy link
Owner Author

caronc commented Sep 2, 2024

@bwachtendorf : Your feedback is incredible; thank you!

I'm not keen on all the overhead invovled with delegate being enabled by defaut as you're the first one to open a ticket on the situation. I am under the impression (but i could very well be wrong 😉 ) that not many people use it.

I did switch the default to be yes in this last PR though. I additionally added the .well-known lookup path first and if that fails, it will check the SRV queries (both the new and old in that order).

I very much value your feedback so please let me know your thoughts.

@bwachtendorf
Copy link

bwachtendorf commented Sep 3, 2024 •
edited
Loading

I just tested it with with following results:

  1. matrixs:// did not work at all
  2. .well-known found the correct domain but completely ignored the result. Logs:
2024-09-03 16:11:11,362 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0824s
2024-09-03 16:11:11,363 - DEBUG - Loaded Matrix URL: matrix://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&delegate=yes&format=text&overflow=upstream
2024-09-03 16:11:11,363 - DEBUG - Matrix POST URL: http://<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-03 16:11:11,363 - DEBUG - Matrix Payload: None
2024-09-03 16:11:11,403 - DEBUG - Matrix Response: code=200, b'{"m.server": "matrix.<domain>:443"}'
2024-09-03 16:11:11,404 - DEBUG - Matrix .well-known resolved to matrix.<domain>:443
2024-09-03 16:11:11,404 - DEBUG - Matrix POST URL: http://<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-03 16:11:11,404 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-03 16:11:11,435 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:11:11,435 - WARNING - Invalid response from Matrix server.
2024-09-03 16:11:11,435 - DEBUG - Response Details:
b''
2024-09-03 16:11:11,435 - DEBUG - Matrix POST URL: http://<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-03 16:11:11,435 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<username>', 'password': '<password>'}
2024-09-03 16:11:11,465 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:11:11,465 - WARNING - Invalid response from Matrix server.
2024-09-03 16:11:11,465 - DEBUG - Response Details:
b''
  1. SRV lookup fails despite there being a SRV record. Logs:
2024-09-03 16:51:40,145 - DEBUG - Loaded Matrix URL: matrix://<user>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&delegate=yes&format=text&overflow=upstream
2024-09-03 16:51:40,145 - DEBUG - Matrix POST URL: http://<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-03 16:51:40,145 - DEBUG - Matrix Payload: None
2024-09-03 16:51:40,198 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:51:40,198 - WARNING - Invalid response from Matrix server.
2024-09-03 16:51:40,198 - DEBUG - Response Details:
b''
2024-09-03 16:51:40,216 - WARNING - Matrix _matrix-fed._tcp.<domain> SRV query failed
2024-09-03 16:51:40,216 - DEBUG - Matrix SRV Exception: [Errno -2] Name or service not known
2024-09-03 16:51:40,236 - WARNING - Matrix _matrix._tcp.<domain> SRV query failed
2024-09-03 16:51:40,236 - DEBUG - Matrix SRV Exception: [Errno -5] No address associated with hostname
2024-09-03 16:51:40,236 - DEBUG - Matrix POST URL: http://<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-03 16:51:40,236 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<user>'}, 'password': '<password>'}
2024-09-03 16:51:40,266 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:51:40,266 - WARNING - Invalid response from Matrix server.
2024-09-03 16:51:40,266 - DEBUG - Response Details:
b''
2024-09-03 16:51:40,266 - DEBUG - Matrix POST URL: http://<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-03 16:51:40,266 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<user>', 'password': '<password>'}
2024-09-03 16:51:40,290 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:51:40,290 - WARNING - Invalid response from Matrix server.
2024-09-03 16:51:40,290 - DEBUG - Response Details:
b''

dig result:

dig -t srv _matrix._tcp.<domain>

; <<>> DiG 9.20.1 <<>> -t srv _matrix._tcp.<domain>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47150
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_matrix._tcp.<domain>.  IN      SRV

;; ANSWER SECTION:
_matrix._tcp.<domain>. 86050 IN  SRV     0 0 443 matrix.<domain>.

;; Query time: 6 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Tue Sep 03 16:28:27 CEST 2024
;; MSG SIZE  rcvd: 99
  1. It's still working when directly providing the correct domain, but it tries to delegate when it already has the correct domain. Logs:
2024-09-03 16:57:28,135 - DEBUG - Loaded Matrix URL: matrix://<user>:****@matrix.<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&delegate=yes&format=text&overflow=upstream
2024-09-03 16:57:28,136 - DEBUG - Matrix POST URL: http://matrix.<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-03 16:57:28,136 - DEBUG - Matrix Payload: None
2024-09-03 16:57:28,187 - DEBUG - Matrix Response: code=200, b''
2024-09-03 16:57:28,187 - WARNING - Invalid response from Matrix server.
2024-09-03 16:57:28,187 - DEBUG - Response Details:
b''
2024-09-03 16:57:28,220 - WARNING - Matrix _matrix-fed._tcp.matrix.<domain> SRV query failed
2024-09-03 16:57:28,220 - DEBUG - Matrix SRV Exception: [Errno -2] Name or service not known
2024-09-03 16:57:28,237 - WARNING - Matrix _matrix._tcp.matrix.<domain> SRV query failed
2024-09-03 16:57:28,237 - DEBUG - Matrix SRV Exception: [Errno -2] Name or service not known
2024-09-03 16:57:28,237 - DEBUG - Matrix POST URL: http://matrix.<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-03 16:57:28,237 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<user>'}, 'password': '<password>'}
2024-09-03 16:57:28,525 - DEBUG - Matrix Response: code=200, b'{"user_id":"@<user>:<domain>","access_token":"<token>","home_server":"<domain>","device_id":"<device_id>"}'
2024-09-03 16:57:28,525 - DEBUG - Authenticated successfully with Matrix server.
2024-09-03 16:57:28,525 - DEBUG - Matrix POST URL: http://matrix.<domain>/_matrix/client/v3/join/<room_id>%3A<domain> (cert_verify=True)
2024-09-03 16:57:28,525 - DEBUG - Matrix Payload: {}
2024-09-03 16:57:28,600 - DEBUG - Matrix Response: code=200, b'{"room_id":"<room_id>:<domain>"}'
2024-09-03 16:57:28,600 - DEBUG - Matrix POST URL: http://matrix.<domain>/_matrix/client/v3/rooms/<room_id>%3A<domain>/send/m.room.message/0 (cert_verify=True)
2024-09-03 16:57:28,601 - DEBUG - Matrix Payload: {'msgtype': 'm.text', 'body': 'test'}
2024-09-03 16:57:28,704 - DEBUG - Matrix Response: code=200, b'{"event_id":"$lNe0KMOn5f_Hvb9L8U_BeC2Fc5kGiF8-s7toPx2jNbE"}'

Maybe you could query https://<domain>/_matrix/federation/v1/version to check if the first given domain is a valid matrix server

@caronc
Copy link
Owner Author

caronc commented Sep 8, 2024

@bwachtendorf I didn't forget about this request; it just turned out to be so much bigger then i thought it would be.

I very heavily added inline documentation to my code to make it easy to follow. Inline documentation follows the link you shared.

I think I fixed the parsing for m.server, and as I understand it, it is the Host Header value i'm setting based off of all of the rules.

This last commit is a complete refactor of everthing. delegate=yes renamed to discovery=yes . Other things to note,

  • m.sever handling and IP Lookups work with native build of apprise.
  • SRV requests as it turns out are not supported at all by native python without a lot of cumbersome and error-prone code. So Apprise will leverage dnspython if it's additionally installed, otherwise lookups all stop after the m.server GET request.

I'm not sure if this works yet as i can't really test (i don't have a matrix server anymore, but i'll start researching how to get one going again).

In the mean-time i still pulled off 100% test coverage without disrupting the working code and their testing... hence there is a good chance this may work (and if not, there won't be too much left to update)... i hope 🙏

I also want to add that i really appreciate your testing and all your great feedback after my last commit 🚀

@bwachtendorf
Copy link

So just some quick tests because it's late here:

  1. matrixs:// works now
  2. When providing correct subdomain directly ?discovery=no has to be used because otherwise it's currently not working at all, because it tries to find .well-known and SRV twice and fails both times. Logs:
2024-09-08 23:04:48,643 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0730s
2024-09-08 23:04:48,643 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@matrix.<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-08 23:04:48,643 - DEBUG - Matrix POST URL: https://matrix.<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 23:04:48,643 - DEBUG - Matrix Payload: None
2024-09-08 23:04:48,649 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:04:48,649 - WARNING - Invalid response from Matrix server.
2024-09-08 23:04:48,649 - DEBUG - Response Details:
b''
2024-09-08 23:04:48,649 - WARNING - Matrix matrix.<domain> SRV lookup not possible until "dnspython" is installed; further delegation processing skipped
2024-09-08 23:04:48,650 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-08 23:04:48,650 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-08 23:04:48,653 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:04:48,653 - WARNING - Invalid response from Matrix server.
2024-09-08 23:04:48,653 - DEBUG - Response Details:
b''
2024-09-08 23:04:48,653 - DEBUG - Matrix POST URL: https://matrix.<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 23:04:48,653 - DEBUG - Matrix Payload: None
2024-09-08 23:04:48,657 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:04:48,657 - WARNING - Invalid response from Matrix server.
2024-09-08 23:04:48,657 - DEBUG - Response Details:
b''
2024-09-08 23:04:48,657 - WARNING - Matrix matrix.<domain> SRV lookup not possible until "dnspython" is installed; further delegation processing skipped
2024-09-08 23:04:48,658 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-08 23:04:48,658 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<username>', 'password': '<password>'}
2024-09-08 23:04:48,661 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:04:48,661 - WARNING - Invalid response from Matrix server.
2024-09-08 23:04:48,661 - DEBUG - Response Details:
b''
  1. .well-known is still being ignored. Logs:
2024-09-08 22:52:19,449 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0798s
2024-09-08 22:52:19,449 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-08 22:52:19,449 - DEBUG - Matrix POST URL: https://<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 22:52:19,449 - DEBUG - Matrix Payload: None
2024-09-08 22:52:19,454 - DEBUG - Matrix Response: code=200, b'{"m.server": "matrix.<domain>:443"}'
2024-09-08 22:52:19,455 - DEBUG - Matrix POST URL: https://<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-08 22:52:19,455 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-08 22:52:19,459 - DEBUG - Matrix Response: code=200, b''
2024-09-08 22:52:19,459 - WARNING - Invalid response from Matrix server.
2024-09-08 22:52:19,459 - DEBUG - Response Details:
b''
2024-09-08 22:52:19,459 - DEBUG - Matrix POST URL: https://<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-08 22:52:19,459 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<username>', 'password': '<password>'}
2024-09-08 22:52:19,463 - DEBUG - Matrix Response: code=200, b''
2024-09-08 22:52:19,463 - WARNING - Invalid response from Matrix server.
2024-09-08 22:52:19,463 - DEBUG - Response Details:
b''
  1. Installing dnspython breaks it completely. Logs:
2024-09-08 22:55:20,534 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0733s
2024-09-08 22:55:20,534 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-08 22:55:20,534 - DEBUG - Matrix POST URL: https://<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 22:55:20,534 - DEBUG - Matrix Payload: None
2024-09-08 22:55:20,562 - DEBUG - Matrix Response: code=200, b''
2024-09-08 22:55:20,562 - WARNING - Invalid response from Matrix server.
2024-09-08 22:55:20,562 - DEBUG - Response Details:
b''
2024-09-08 22:55:20,562 - ERROR - Unhandled Notification Exception
Traceback (most recent call last):
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/apprise.py", line 582, in _notify_sequential
    result = server.notify(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/base.py", line 423, in notify
    the_calls = [self.send(**kwargs2) for kwargs2 in send_calls]
                 ^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 425, in send
    return getattr(self, '_send_{}_notification'.format(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 630, in _send_server_notification
    if not self._login():
           ^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 933, in _login
    postokay, response = self._fetch('/login', payload=payload)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 1249, in _fetch
    host = self.server_discovery()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 1802, in server_discovery
    _host = self.__srv_lookup(f'_matrix-fed._tcp.{host}')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<repo_dir>/.venv/lib/python3.12/site-packages/apprise/plugins/matrix.py", line 1852, in __srv_lookup
    resolver = dns.resolver.Resolver()
               ^^^^^^^^^^^^
AttributeError: module 'dns' has no attribute 'resolver'

@bwachtendorf
Copy link

With the latest changes:

  1. still applies partially. Logs:
2024-09-08 23:23:23,745 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0786s
2024-09-08 23:23:23,746 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@matrix.<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-08 23:23:23,746 - DEBUG - Matrix POST URL: https://matrix.<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 23:23:23,746 - DEBUG - Matrix Payload: None
2024-09-08 23:23:23,751 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:23:23,751 - WARNING - Invalid response from Matrix server.
2024-09-08 23:23:23,751 - DEBUG - Response Details:
b''
2024-09-08 23:23:23,751 - WARNING - Matrix matrix.<domain> SRV lookup not possible until "dnspython" is installed; further delegation processing skipped
2024-09-08 23:23:23,752 - DEBUG - socket.getaddrinfo matrix.<domain> resolved to <correct_host_ip>
2024-09-08 23:23:23,752 - DEBUG - Matrix SD - Resolved with DNS Lookup Rule: <correct_host_ip>:8448
2024-09-08 23:23:23,752 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-08 23:23:23,752 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-08 23:23:23,755 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:23:23,755 - WARNING - Invalid response from Matrix server.
2024-09-08 23:23:23,755 - DEBUG - Response Details:
b''
2024-09-08 23:23:23,755 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-08 23:23:23,756 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<username>', 'password': '<password>'}
2024-09-08 23:23:23,759 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:23:23,759 - WARNING - Invalid response from Matrix server.
2024-09-08 23:23:23,759 - DEBUG - Response Details:
b''
  1. still applies:
2024-09-08 23:20:00,206 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0727s
2024-09-08 23:20:00,206 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-08 23:20:00,206 - DEBUG - Matrix POST URL: https://<domain>/.well-known/matrix/server (cert_verify=True)
2024-09-08 23:20:00,206 - DEBUG - Matrix Payload: None
2024-09-08 23:20:00,212 - DEBUG - Matrix Response: code=200, b'{"m.server": "matrix.<domain>:443"}'
2024-09-08 23:20:00,213 - DEBUG - socket.getaddrinfo matrix.<domain> resolved to <correct_host_ip>
2024-09-08 23:20:00,213 - DEBUG - Matrix SD - Resolved with .well-known Port Rule: <correct_host_ip>:443
2024-09-08 23:20:00,213 - DEBUG - Matrix POST URL: https://<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-08 23:20:00,213 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-08 23:20:00,217 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:20:00,217 - WARNING - Invalid response from Matrix server.
2024-09-08 23:20:00,217 - DEBUG - Response Details:
b''
2024-09-08 23:20:00,217 - DEBUG - Matrix POST URL: https://<domain>/_matrix/client/v3/register (cert_verify=True)
2024-09-08 23:20:00,217 - DEBUG - Matrix Payload: {'kind': 'user', 'auth': {'type': 'm.login.dummy'}, 'username': '<username>', 'password': '<password>'}
2024-09-08 23:20:00,220 - DEBUG - Matrix Response: code=200, b''
2024-09-08 23:20:00,220 - WARNING - Invalid response from Matrix server.
2024-09-08 23:20:00,220 - DEBUG - Response Details:
b''
  1. still applies

@caronc
Copy link
Owner Author

caronc commented Sep 8, 2024

Enjoy your sleep! 🙏

I think i'm going to need to work offline with you a bit here. Can you send me an email to [email protected] and/or log into Discord?

This one has me baffled as the GET request should work. This is one thing i'd like to work with you offline with. I'm hoping you can share your domain with me (if it's not to much trouble and/or an invasion of privacy). I'd like to just test against your setup.

If the matrixs://domain/ ends up being the same domain found in the m.server or SRV lookup, should i just ignore the delegation completely then?

@caronc caronc changed the title Matrix Server Delegation Support (m.server param) Matrix Well Known URI Server Discovery Sep 14, 2024
@caronc
Copy link
Owner Author

caronc commented Sep 14, 2024

@bwachtendorf I think I got it. Give it another go and let me know.

@bwachtendorf
Copy link

First off. It works!
But there are still some minor problems.

  1. It does a lot of requests that are not necessary:
2024-09-14 22:39:04,984 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0733s
2024-09-14 22:39:04,985 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-14 22:39:04,985 - DEBUG - Matrix GET URL: https://<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:39:04,985 - DEBUG - Matrix Payload: None
2024-09-14 22:39:04,989 - DEBUG - Matrix Response: code=200, b'{"m.<username>": {"base_url": "https://matrix.<domain>"}}'
2024-09-14 22:39:04,990 - DEBUG - Matrix GET URL: https://matrix.<domain>/_matrix/client/versions (cert_verify=True)
2024-09-14 22:39:04,990 - DEBUG - Matrix Payload: None
2024-09-14 22:39:04,995 - DEBUG - Matrix Response: code=200, b'{"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0","r0.6.1","v1.1","v1.2","v1.3","v1.4","v1.5","v1.6","v1.7","v1.8","v1.9","v1.10","v1.11"],"unstable_features":{"org.matrix.label_based_filtering":true,"org.matrix.e2e_cross_signing":true,"org.matrix.msc2432":true,"uk.half-shot.msc2666.query_mutual_rooms":true,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.msc3026.busy_presence":false,"org.matrix.msc2285.stable":true,"org.matrix.msc3827.stable":true,"org.matrix.msc3440.stable":true,"org.matrix.msc3771":true,"org.matrix.msc3773":false,"fi.mau.msc2815":false,"fi.mau.msc2659.stable":true,"org.matrix.msc3882":false,"org.matrix.msc3881":false,"org.matrix.msc3874":false,"org.matrix.msc3886":false,"org.matrix.msc3912":false,"org.matrix.msc3981":true,"org.matrix.msc3391":false,"org.matrix.msc4069":false,"org.matrix.msc4028":false,"org.matrix.msc4108":false,"org.matrix.msc4151":false,"org.matrix.simplified_msc3575":true}}'
2024-09-14 22:39:04,995 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-14 22:39:04,995 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-14 22:39:05,254 - DEBUG - Matrix Response: code=200, b'{"user_id":"@<username>:<domain>","access_token":"<access_token>","home_server":"<domain>","device_id":"GRHSKNBHZA"}'
2024-09-14 22:39:05,254 - DEBUG - Authenticated successfully with Matrix server.
2024-09-14 22:39:05,254 - DEBUG - Matrix GET URL: https://<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:39:05,255 - DEBUG - Matrix Payload: None
2024-09-14 22:39:05,258 - DEBUG - Matrix Response: code=200, b'{"m.<username>": {"base_url": "https://matrix.<domain>"}}'
2024-09-14 22:39:05,259 - DEBUG - Matrix GET URL: https://matrix.<domain>/_matrix/client/versions (cert_verify=True)
2024-09-14 22:39:05,259 - DEBUG - Matrix Payload: None
2024-09-14 22:39:05,268 - DEBUG - Matrix Response: code=200, b'{"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0","r0.6.1","v1.1","v1.2","v1.3","v1.4","v1.5","v1.6","v1.7","v1.8","v1.9","v1.10","v1.11"],"unstable_features":{"org.matrix.label_based_filtering":true,"org.matrix.e2e_cross_signing":true,"org.matrix.msc2432":true,"uk.half-shot.msc2666.query_mutual_rooms":true,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.msc3026.busy_presence":false,"org.matrix.msc2285.stable":true,"org.matrix.msc3827.stable":true,"org.matrix.msc3440.stable":true,"org.matrix.msc3771":true,"org.matrix.msc3773":false,"fi.mau.msc2815":false,"fi.mau.msc2659.stable":true,"org.matrix.msc3882":false,"org.matrix.msc3881":false,"org.matrix.msc3874":false,"org.matrix.msc3886":false,"org.matrix.msc3912":false,"org.matrix.msc3981":true,"org.matrix.msc3391":false,"org.matrix.msc4069":false,"org.matrix.msc4028":false,"org.matrix.msc4108":false,"org.matrix.msc4151":false,"org.matrix.simplified_msc3575":true}}'
2024-09-14 22:39:05,268 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/join/<room_id>%3A<domain> (cert_verify=True)
2024-09-14 22:39:05,268 - DEBUG - Matrix Payload: {}
2024-09-14 22:39:05,293 - DEBUG - Matrix Response: code=200, b'{"room_id":"<room_id>:<domain>"}'
2024-09-14 22:39:05,293 - DEBUG - Matrix GET URL: https://<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:39:05,293 - DEBUG - Matrix Payload: None
2024-09-14 22:39:05,297 - DEBUG - Matrix Response: code=200, b'{"m.<username>": {"base_url": "https://matrix.<domain>"}}'
2024-09-14 22:39:05,297 - DEBUG - Matrix GET URL: https://matrix.<domain>/_matrix/client/versions (cert_verify=True)
2024-09-14 22:39:05,297 - DEBUG - Matrix Payload: None
2024-09-14 22:39:05,303 - DEBUG - Matrix Response: code=200, b'{"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0","r0.6.1","v1.1","v1.2","v1.3","v1.4","v1.5","v1.6","v1.7","v1.8","v1.9","v1.10","v1.11"],"unstable_features":{"org.matrix.label_based_filtering":true,"org.matrix.e2e_cross_signing":true,"org.matrix.msc2432":true,"uk.half-shot.msc2666.query_mutual_rooms":true,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.msc3026.busy_presence":false,"org.matrix.msc2285.stable":true,"org.matrix.msc3827.stable":true,"org.matrix.msc3440.stable":true,"org.matrix.msc3771":true,"org.matrix.msc3773":false,"fi.mau.msc2815":false,"fi.mau.msc2659.stable":true,"org.matrix.msc3882":false,"org.matrix.msc3881":false,"org.matrix.msc3874":false,"org.matrix.msc3886":false,"org.matrix.msc3912":false,"org.matrix.msc3981":true,"org.matrix.msc3391":false,"org.matrix.msc4069":false,"org.matrix.msc4028":false,"org.matrix.msc4108":false,"org.matrix.msc4151":false,"org.matrix.simplified_msc3575":true}}'
2024-09-14 22:39:05,303 - DEBUG - Matrix <function put at 0x7fe5cdea36a0> URL: https://matrix.<domain>/_matrix/client/v3/rooms/<room_id>%3A<domain>/send/m.room.message/0 (cert_verify=True)
2024-09-14 22:39:05,303 - DEBUG - Matrix Payload: {'msgtype': 'm.text', 'body': 'test'}
2024-09-14 22:39:05,353 - DEBUG - Matrix Response: code=200, b'{"event_id":"$EnvXRoOiXIBdodMw9yzKxYl9kv431A2b0yggnsDSn7I"}'

I got rate limited by my own server quickly while testing:

2024-09-14 22:39:08,251 - DEBUG - Matrix Response: code=429, b'{"errcode":"M_LIMIT_EXCEEDED","error":"Too Many Requests","retry_after_ms":205563}'
2024-09-14 22:39:08,251 - WARNING - Matrix server requested we throttle back 205.563ms; retries left 1.
  1. Using the correct domain directly requires ?discovery=no:
    With discovery:
2024-09-14 22:38:13,207 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0758s
2024-09-14 22:38:13,207 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@matrix.<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-14 22:38:13,207 - DEBUG - Matrix GET URL: https://matrix.<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:38:13,207 - DEBUG - Matrix Payload: None
2024-09-14 22:38:13,214 - DEBUG - Matrix Response: code=200, b''
2024-09-14 22:38:13,214 - WARNING - Invalid response from Matrix server.
2024-09-14 22:38:13,214 - DEBUG - Response Details:
b''
2024-09-14 22:38:13,214 - WARNING - Matrix Well-Known Base URI Discovery Failed - https://matrix.<domain>/.well-known/matrix/client returned error code: 500
2024-09-14 22:38:13,214 - DEBUG - Matrix GET URL: https://matrix.<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:38:13,214 - DEBUG - Matrix Payload: None
2024-09-14 22:38:13,219 - DEBUG - Matrix Response: code=200, b''
2024-09-14 22:38:13,219 - WARNING - Invalid response from Matrix server.
2024-09-14 22:38:13,219 - DEBUG - Response Details:
b''
2024-09-14 22:38:13,219 - WARNING - Matrix Well-Known Base URI Discovery Failed - https://matrix.<domain>/.well-known/matrix/client returned error code: 500

Without:

2024-09-14 22:38:47,166 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0750s
2024-09-14 22:38:47,167 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@matrix.<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=no&format=text&overflow=upstream
2024-09-14 22:38:47,167 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/login (cert_verify=True)
2024-09-14 22:38:47,167 - DEBUG - Matrix Payload: {'type': 'm.login.password', 'identifier': {'type': 'm.id.user', 'user': '<username>'}, 'password': '<password>'}
2024-09-14 22:38:47,432 - DEBUG - Matrix Response: code=200, b'{"user_id":"@<username>:<domain>","access_token":"<access_token>","home_server":"<domain>","device_id":"ZZISDVABGS"}'
2024-09-14 22:38:47,432 - DEBUG - Authenticated successfully with Matrix server.
2024-09-14 22:38:47,432 - DEBUG - Matrix POST URL: https://matrix.<domain>/_matrix/client/v3/join/<room_id>%3A<domain> (cert_verify=True)
2024-09-14 22:38:47,432 - DEBUG - Matrix Payload: {}
2024-09-14 22:38:47,459 - DEBUG - Matrix Response: code=200, b'{"room_id":"<room_id>:<domain>"}'
2024-09-14 22:38:47,460 - DEBUG - Matrix <function put at 0x7e13c02bf740> URL: https://matrix.<domain>/_matrix/client/v3/rooms/<room_id>%3A<domain>/send/m.room.message/0 (cert_verify=True)
2024-09-14 22:38:47,460 - DEBUG - Matrix Payload: {'msgtype': 'm.text', 'body': 'test'}
2024-09-14 22:38:47,512 - DEBUG - Matrix Response: code=200, b'{"event_id":"$A4iByAeZf2nYXamtff3o_LFcTMYJoTONENOphDFU7RI"}'
  1. Caching does not seem to work correctly because apprise still tries to discover the server that was just discovered, instead of just skipping to the PUT query:
2024-09-14 22:43:31,436 - DEBUG - Notification Plugin 109(s) and 155 Schema(s) loaded in 0.0766s
2024-09-14 22:43:31,437 - DEBUG - Loaded Matrix URL: matrixs://<username>:****@<domain>/<room_id>?image=no&mode=off&version=3&msgtype=text&discovery=yes&format=text&overflow=upstream
2024-09-14 22:43:31,437 - DEBUG - Matrix GET URL: https://<domain>/.well-known/matrix/client (cert_verify=True)
2024-09-14 22:43:31,437 - DEBUG - Matrix Payload: None
2024-09-14 22:43:31,442 - DEBUG - Matrix Response: code=200, b'{"m.<username>": {"base_url": "https://matrix.<domain>"}}'
2024-09-14 22:43:31,443 - DEBUG - Matrix GET URL: https://matrix.<domain>/_matrix/client/versions (cert_verify=True)
2024-09-14 22:43:31,443 - DEBUG - Matrix Payload: None
2024-09-14 22:43:31,448 - DEBUG - Matrix Response: code=200, b'{"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0","r0.6.1","v1.1","v1.2","v1.3","v1.4","v1.5","v1.6","v1.7","v1.8","v1.9","v1.10","v1.11"],"unstable_features":{"org.matrix.label_based_filtering":true,"org.matrix.e2e_cross_signing":true,"org.matrix.msc2432":true,"uk.half-shot.msc2666.query_mutual_rooms":true,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.msc3026.busy_presence":false,"org.matrix.msc2285.stable":true,"org.matrix.msc3827.stable":true,"org.matrix.msc3440.stable":true,"org.matrix.msc3771":true,"org.matrix.msc3773":false,"fi.mau.msc2815":false,"fi.mau.msc2659.stable":true,"org.matrix.msc3882":false,"org.matrix.msc3881":false,"org.matrix.msc3874":false,"org.matrix.msc3886":false,"org.matrix.msc3912":false,"org.matrix.msc3981":true,"org.matrix.msc3391":false,"org.matrix.msc4069":false,"org.matrix.msc4028":false,"org.matrix.msc4108":false,"org.matrix.msc4151":false,"org.matrix.simplified_msc3575":true}}'
2024-09-14 22:43:31,448 - DEBUG - Matrix <function put at 0x7dee2048f6a0> URL: https://matrix.<domain>/_matrix/client/v3/rooms/<room_id>%3A<domain>/send/m.room.message/2 (cert_verify=True)
2024-09-14 22:43:31,448 - DEBUG - Matrix Payload: {'msgtype': 'm.text', 'body': 'test'}
2024-09-14 22:43:31,527 - DEBUG - Matrix Response: code=200, b'{"event_id":"$FpH-IGZjKzbBhDWZhIDC-E1zeWfdgiask-udxH24oDI"}'

In my opinion the process should look like this:

  1. Check cache
    1. If we hit cache continue with 6
    2. If we don't hit cache continue with 2
  2. Check .well-known/matrix/client
    1. If 200 with content continue with 3
    2. Everything else continue with 4
  3. Read delegation info from .well-known/matrix/client
  4. Verify matrix server with _matrix/client/versions
  5. Join room if not joined
  6. Post message

That would be max. 5 requests and not 9 like it is now. When hitting something in the cache it should require just 1 request and not 3.

@caronc
Copy link
Owner Author

caronc commented Sep 14, 2024 via email

@caronc
Copy link
Owner Author

caronc commented Sep 14, 2024 via email

@bwachtendorf
Copy link

I don't see any more problems on my end. Everything works now as I would expect it.
Fantastic work!

@caronc
Copy link
Owner Author

caronc commented Sep 16, 2024

Before i merge; is it worth bumping up the cache to say 30 days before searching for the well-known entry again, or leave it at 1 day ? I just don't want to create more web hits than necessary. Thoughts?

@bwachtendorf
Copy link

I think 24 hours is OK, otherwise we might run into problems, where the cache has to be manually deleted because of old values.

@caronc caronc merged commit f6c48d0 into master Sep 16, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Matrix delegation not working
2 participants
@caronc @bwachtendorf