Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

cbirchinger · 2024-09-12T18:37:16Z

After updating from 0.19.2 to 0.19.3 the server rejects the connection with 401 instead of 207 (-vdebug outputs below)

Nothing other than the update from 0.19.2 to 0.19.3 was changed. Same Python, same configuration, same server setup. Downgrading to 0.19.2 again fixes the issue and the sync works again

Server: Radicale 3.2.3 behind Apache 2.4.62 (mod_proxy)
OS: Gentoo Linux (Server and Client)
Python: 3.12

Hardcoded password instead of password.fetch was also tried without success.

-vdebug output of 0.19.3 (FAIL)

debug: Fetching value for password.fetch with command strategy.
debug: Found cached value for ['command', '~/.config/vdirsyncer/pass.sh'].
Syncing user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
debug: 401
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:44:17 GMT', 'Server': 'Apache', 'WWW-Authenticate': 'Basic realm="Radicale - Password Required"', 'Content-Length': '448', 'Content-Type': 'text/html; charset=iso-8859-1')>
debug: <StreamReader 448 bytes eof>
error: Unknown error occurred for user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX: 401, message='Unauthorized', url='https://www.domain.xxx/radicale/joker/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/cli/tasks.py", line 74, in sync_collection
debug:     await sync.sync(
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/sync/__init__.py", line 150, in sync
debug:     b_nonempty = await b_info.prepare_new_status()
debug:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/sync/__init__.py", line 55, in prepare_new_status
debug:     async for href, etag in self.storage.list():  # type: ignore[attr-defined]
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/storage/dav.py", line 672, in list
debug:     response = await self.session.request(
debug:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/storage/dav.py", line 413, in request
debug:     return await http.request(method, url, session=session, **more)
debug:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/http.py", line 216, in request
debug:     response.raise_for_status()
debug:   File "/usr/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1093, in raise_for_status
debug:     raise ClientResponseError(
debug: 401
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:44:17 GMT', 'Server': 'Apache', 'WWW-Authenticate': 'Basic realm="Radicale - Password Required"', 'Content-Length': '448', 'Content-Type': 'text/html; charset=iso-8859-1')>
debug: <StreamReader 448 bytes eof>
error: Unknown error occurred for user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX: 401, message='Unauthorized', url='https://www.domain.xxx/radicale/joker/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'
error: Use `-vdebug` to see the full traceback.

-vdebug output of 0.19.2 (OK)

debug: Fetching value for password.fetch with command strategy.
debug: Found cached value for ['command', '~/.config/vdirsyncer/pass.sh'].
Syncing user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
debug: 207
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:43:26 GMT', 'Server': 'WSGIServer/0.2 CPython/3.12.3', 'DAV': '1, 2, 3, calendar-access, addressbook, extended-mkcol', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Encoding': 'gzip', 'Content-Length': '1132')>
debug: <StreamReader>
debug: Already normalized: '/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'

config

[general]
status_path = "~/.local/share/vdirsyncer/"

# ---[ CardDav contacts sync ]--------------------------------------------------------
[pair user_contacts]
a = "user_contacts_local"
b = "user_contacts_remote"
collections = ["XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX"]
metadata = ["displayname"]

[storage user_contacts_local]
type = "filesystem"
path = "~/.contacts/"
fileext = ".vcf"

[storage user_contacts_remote]
type = "carddav"
url = "https://www.domain.xxx/radicale/user/"
username = "user"
password.fetch = ["command", "~/.config/vdirsyncer/pass.sh"]
# -------------------------------------------------------------------------------------

# ---[ CalDav calendar sync ]---------------------------------------------------------
[pair user_calendar]
a = "user_calendar_local"
b = "user_calendar_remote"
collections = ["XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX", "XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX"]
metadata = ["displayname", "color"]

[storage user_calendar_local]
type = "filesystem"
path = "~/.calendars/"
fileext = ".ics"

[storage user_calendar_remote]
type = "caldav"
url = "https://www.domain.xxx/radicale/user/"
username = "user"
password.fetch = ["command", "~/.config/vdirsyncer/pass.sh"]
# -------------------------------------------------------------------------------------

Apache vhost

RewriteEngine On
RewriteRule ^/radicale$ /radicale/ [R,L]

<Location "/radicale/">
    <IfModule security2_module>
        SecAuditEngine Off
        SecRuleEngine Off
    </IfModule>

    AuthType     Basic
    AuthName     "Radicale - Password Required"
    AuthUserFile "/etc/radicale/users"
    Require      valid-user

    ProxyPass        http://localhost:5232/ retry=0
    ProxyPassReverse http://localhost:5232/
    RequestHeader    set X-Script-Name /radicale
    RequestHeader    set X-Remote-User expr=%{REMOTE_USER}
</Location>

The text was updated successfully, but these errors were encountered:

WhyNotHugo · 2024-09-13T16:35:19Z

cc: @malmeloo

malmeloo · 2024-09-13T18:56:09Z

I did a little digging and what I think is happening here is aiohttp dropping the Authorization header when it encounters a redirect. Could you post a few log lines from Apache to confirm that that's also what's going on here?

cbirchinger · 2024-09-13T20:36:54Z

Ah, aiohttp! I think i've found the issue.

Changed in version 3.0: Added support for ~/.netrc file.
Changed in version 3.9: Added support for reading HTTP Basic Auth credentials from ~/.netrc file.

It is using a credential entry in ~/.netrc for the autentication. Because in ~/.netrc only the machine name is configured so it seems to be using it for all requests on that host regardless of the subpath.

The question now is can it be changed so the values in ~/.config/vdirsyncer/config have higher priority than the generic ~/.netrc

EDIT: As far as i understand the aiohttp documentation ~/.netrc should only be used related to proxy authentication. Does it consider a web service behind a reverse proxy such as Apache or nginx needs to be treated like a Squid web cache that requires auth?

WhyNotHugo · 2024-09-14T07:02:23Z

I’ll take a patch that disables netrc support when invoking aiohttp. It isn’t a supported mechanism for Vdirsyncer.

Hawk777 · 2024-09-16T05:48:37Z

aiohttp 3.9 added support for using netrc for non-proxy auth.

malmeloo · 2024-09-16T08:12:41Z

That option should still be off by default though, so I'm not entirely convinced that's the problem here. But I will investigate a little more and see if I can repro.

cbirchinger · 2024-09-16T08:13:21Z

I don't see an explicit option for this. It's tied to the "trust_env" option in "aiohttp.ClientSession" though

        if auth is None and trust_env and self.url.host is not None:
            netrc_obj = netrc_from_env()
            with contextlib.suppress(LookupError):
                auth = basicauth_from_netrc(netrc_obj, self.url.host)

vdirsyncer currently is using "True":
https://github.com/search?q=repo%3Apimutils%2Fvdirsyncer%20trust_env&type=code

Setting it to "False" fixes the issue for me. It also does disable a users ability to set the NETRC environment to have an alternative to the hardcoded ~/.netrc

malmeloo · 2024-09-16T08:17:56Z

Oh the option is enabled? Right, that explains the issue then. I think disabling it would also disable support for setting proxy details using env vars though, is that desirable?

cbirchinger · 2024-09-16T08:35:00Z

Not really, a lot more people probably use the "http_proxy" environment variables.

Knowing the root cause of the issue, I have some workarounds for the time being like running "NETRC=/dev/null vdirsyncer sync"

malmeloo · 2024-09-16T09:11:31Z

If that fixes it we could probably override that env var in vdirsyncer, as a workaround...

The more I learn about aiohttp, the more I'm starting to dislike it.

cbirchinger · 2024-09-16T10:53:32Z

Yes, please. Having the env var set to something like /dev/null by default would help. vdirsyncer has it's own username and password settings. I see no benefit in using the less flexible and if present probably used for something else ~/.netrc instead.

malmeloo · 2024-09-17T21:25:39Z

@cbirchinger could you try #1141 to see if it fixes the issue?

cbirchinger · 2024-09-17T21:46:36Z

@cbirchinger could you try #1141 to see if it fixes the issue?

Yes, that fixes the issue. Thank you.

malmeloo linked a pull request Sep 17, 2024 that will close this issue

Do not load netrc config files #1141

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

cbirchinger commented Sep 12, 2024

WhyNotHugo commented Sep 13, 2024

malmeloo commented Sep 13, 2024

cbirchinger commented Sep 13, 2024 •

edited

Loading

WhyNotHugo commented Sep 14, 2024 via email

Hawk777 commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 17, 2024

cbirchinger commented Sep 17, 2024

Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

Comments

cbirchinger commented Sep 12, 2024

WhyNotHugo commented Sep 13, 2024

malmeloo commented Sep 13, 2024

cbirchinger commented Sep 13, 2024 • edited Loading

WhyNotHugo commented Sep 14, 2024 via email

Hawk777 commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 16, 2024

cbirchinger commented Sep 16, 2024

malmeloo commented Sep 17, 2024

cbirchinger commented Sep 17, 2024

cbirchinger commented Sep 13, 2024 •

edited

Loading