Issue aggregating multiple iCalendars feeds into a single CalDAV calendar

[mdrovdahl]

My use case is:

• Aggregate 4 iCalendar distinct feeds (each from a different soccer team) into a single, combined calendar
• In practice this is 3 ICS feeds from TeamSnap + 1 ICS feed from Google Calendar aggregated into a Radicale hosted CalDAV calendar
• For purposes of testing and reproducing, everything is coming from Radicale
• Changes in the 4 ICS feeds should always overwrite conflicting changes in the CalDAV calendar

The problem is (I think) that when an event is removed/modified during the sync operation between the first calendar feed and the aggregated calendar in [pair one], it causes subsequent pairs to experience a "raise exceptions.NotFoundError(href)" error.

Config file:

# An example configuration for vdirsyncer.
#
# Move it to ~/.vdirsyncer/config or ~/.config/vdirsyncer/config and edit it.
# Run `vdirsyncer --help` for CLI usage.
#
# Optional parameters are commented out.
# This file doesn't document all available parameters, see
# http://vdirsyncer.pimutils.org/ for the rest of them.

[general]
# A folder where vdirsyncer can store some metadata about each pair.
status_path = "~/.config/vdirsyncer/status/"

# CALDAV
[pair one]
a = "Combined_Calendar"
b = "Team_One"
collections = null
metadata = ["displayname", "color"]
conflict_resolution = "b wins"

[pair two]
a = "Combined_Calendar"
b = "Team_Two"
collections = null
metadata = ["displayname", "color"]
conflict_resolution = "b wins"

[pair four]
a = "Combined_Calendar"
b = "Team_Three"
collections = null
metadata = ["displayname", "color"]
conflict_resolution = "b wins"

[pair three]
a = "Combined_Calendar"
b = "Team_Four"
collections = null
metadata = ["displayname", "color"]
conflict_resolution = "b wins"

# Locally stored calendar file via Radicale
[storage Combined_Calendar]
type = "filesystem"
path = "~/.var/lib/radicale/collections/collection-root/markd/56551c03-861f-8936-cf73-18cd923b410f/"
fileext = ".ics"

# Remotely subscribed calendar
[storage Team_One]
type = "http"
url = "https://13649.REDACTED.com/public/1b823f97-da9b-f235-2ceb-ebf7d5369aa5/"

# Remotely subscribed calendar
[storage Team_Two]
type = "http"
url = "https://13649.REDACTED.com/public/e48c79bf-5a75-4f6c-3344-2f8e51b5e9ec/"

# Remotely subscribed calendar
[storage Team_Three]
type = "http"
url = "https://13649.REDACTED.com/public/7fb473bd-9946-881e-4ab8-93f57ea6362d/"

# Remotely subscribed calendar
[storage Team_Four]
type = "http"
url = "https://13649.REDACTED.com/public/1e68e172-ccf3-2a2b-9390-61d789a49727/"

• Your vdirsyncer version: 0.19.2
• If applicable, which server software (and which version) you're using: Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
• Your Python version: 3.9
• Your operating system: Raspbian
• Your config file: see above
• Use vdirsyncer -vdebug for debug output. The output is sensitive, but
  please attach at least the last few lines before the error (if applicable),
  censored as necessary. This is almost always the most useful information.

Debug output:

pi@raspberrypi:~ $ vdirsyncer -vdebug sync
Syncing one
debug: ====================
debug: GET https://13649.REDACTED.com/public/1b823f97-da9b-f235-2ceb-ebf7d5369aa5/
debug: {'User-Agent': 'vdirsyncer/0.19.2'}
debug: None
debug: Sending request...
Syncing two
debug: ====================
debug: GET https://13649.REDACTED.com/public/e48c79bf-5a75-4f6c-3344-2f8e51b5e9ec/
debug: {'User-Agent': 'vdirsyncer/0.19.2'}
debug: None
debug: Sending request...
Syncing four
debug: ====================
debug: GET https://13649.REDACTED.com/public/7fb473bd-9946-881e-4ab8-93f57ea6362d/
debug: {'User-Agent': 'vdirsyncer/0.19.2'}
debug: None
debug: Sending request...
Syncing three
debug: ====================
debug: GET https://13649.REDACTED.com/public/1e68e172-ccf3-2a2b-9390-61d789a49727/
debug: {'User-Agent': 'vdirsyncer/0.19.2'}
debug: None
debug: Sending request...
debug: 200
debug: <CIMultiDictProxy('Server': 'nginx/1.18.0', 'Date': 'Wed, 27 Sep 2023 20:31:45 GMT', 'Content-Type': 'text/calendar; charset=utf-8', 'Content-Length': '604', 'Connection': 'keep-alive', 'Last-Modified': 'Wed, 27 Sep 2023 20:30:30 GMT', 'Etag': '"a4a89f311430b5a492ddfb37d0ffe91117e0a9e050d00109298c9f859045a6eb"', 'Content-Disposition': "attachement; filename*=utf-8''Team%20Four.ics", 'Content-Encoding': 'gzip')>
debug: <StreamReader 1055 bytes eof>
Deleting item 1d0b6542710d6371b19bf407ce91d1bf05e5b4cc5302d6babf2096a5fb8583db from Combined_Calendar
Deleting item 578f95767b43b79d27595e9e26ac2940c4395d812f83f690d3e612f2148a86b8 from Combined_Calendar
debug: 200
debug: <CIMultiDictProxy('Server': 'nginx/1.18.0', 'Date': 'Wed, 27 Sep 2023 20:31:45 GMT', 'Content-Type': 'text/calendar; charset=utf-8', 'Content-Length': '570', 'Connection': 'keep-alive', 'Last-Modified': 'Wed, 27 Sep 2023 19:00:39 GMT', 'Etag': '"9f4f17466a6037c19f7acb927d1b786cfaf415e9288df587a51d6ba9ea22eb7e"', 'Content-Disposition': "attachement; filename*=utf-8''Team%20Two.ics", 'Content-Encoding': 'gzip')>
debug: <StreamReader 1006 bytes eof>
Deleting item 1d0b6542710d6371b19bf407ce91d1bf05e5b4cc5302d6babf2096a5fb8583db from Combined_Calendar
error: Unknown error occurred for two: 1d0b6542710d6371b19bf407ce91d1bf05e5b4cc5302d6babf2096a5fb8583db.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/cli/utils.py", line 70, in handle_cli_error
debug:     raise e
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 160, in sync
debug:     await action.run(a_info, b_info, conflict_resolution, partial_sync)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 183, in run
debug:     await self._run_impl(a, b)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 248, in _run_impl
debug:     await self.dest.storage.delete(meta.href, meta.etag)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 22, in inner
debug:     return await f(self, *args, **kwargs)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/filesystem.py", line 163, in delete
debug:     raise exceptions.NotFoundError(href)
debug: 200
debug: <CIMultiDictProxy('Server': 'nginx/1.18.0', 'Date': 'Wed, 27 Sep 2023 20:31:45 GMT', 'Content-Type': 'text/calendar; charset=utf-8', 'Content-Length': '131', 'Connection': 'keep-alive', 'Last-Modified': 'Wed, 27 Sep 2023 20:31:25 GMT', 'Etag': '"339754d31627963fb6965b0ad02fd98e214a1564d7ed4891aaa3f28ed8343290"', 'Content-Disposition': "attachement; filename*=utf-8''Team%20One.ics", 'Content-Encoding': 'gzip')>
debug: <StreamReader 123 bytes eof>
error: one: Storage "Team_One" was completely emptied. If you want to delete ALL entries on BOTH sides, then use `vdirsyncer sync --force-delete one`. Otherwise delete the files for one in your status directory.
debug: 200
debug: <CIMultiDictProxy('Server': 'nginx/1.18.0', 'Date': 'Wed, 27 Sep 2023 20:31:45 GMT', 'Content-Type': 'text/calendar; charset=utf-8', 'Content-Length': '622', 'Connection': 'keep-alive', 'Last-Modified': 'Wed, 27 Sep 2023 19:05:02 GMT', 'Etag': '"5ae87dc1218702675ee31da3e9600fcdaccadfb244a069d1c7806f42bfa544c7"', 'Content-Disposition': "attachement; filename*=utf-8''Team%20Three.ics", 'Content-Encoding': 'gzip')>
debug: <StreamReader 1094 bytes eof>
Deleting item 1d0b6542710d6371b19bf407ce91d1bf05e5b4cc5302d6babf2096a5fb8583db from Combined_Calendar
error: Unknown error occurred for four: 1d0b6542710d6371b19bf407ce91d1bf05e5b4cc5302d6babf2096a5fb8583db.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/cli/utils.py", line 70, in handle_cli_error
debug:     raise e
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 160, in sync
debug:     await action.run(a_info, b_info, conflict_resolution, partial_sync)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 183, in run
debug:     await self._run_impl(a, b)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 248, in _run_impl
debug:     await self.dest.storage.delete(meta.href, meta.etag)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 22, in inner
debug:     return await f(self, *args, **kwargs)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/filesystem.py", line 163, in delete
debug:     raise exceptions.NotFoundError(href)
Deleting item 578f95767b43b79d27595e9e26ac2940c4395d812f83f690d3e612f2148a86b8 from Combined_Calendar
error: Unknown error occurred for four: 578f95767b43b79d27595e9e26ac2940c4395d812f83f690d3e612f2148a86b8.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/cli/utils.py", line 70, in handle_cli_error
debug:     raise e
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 160, in sync
debug:     await action.run(a_info, b_info, conflict_resolution, partial_sync)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 183, in run
debug:     await self._run_impl(a, b)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 248, in _run_impl
debug:     await self.dest.storage.delete(meta.href, meta.etag)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 22, in inner
debug:     return await f(self, *args, **kwargs)
debug:   File "/home/pi/vdirsyncer_env/lib/python3.9/site-packages/vdirsyncer/storage/filesystem.py", line 163, in delete
debug:     raise exceptions.NotFoundError(href)

[WhyNotHugo]

I understand the use case, and the feature that you're asking for would also be of use to me.

I'm currently re-writing vdirsyncer, and the new implementation should allow adding this in future. I don't think it will be soon.

[mdrovdahl]

Appreciate the swift reply...I've seen your rewrite effort and tinkered a bit with davcli. Best of luck with that effort.

Meanwhile, I may keep poking at this aggregation use case b/c it's fundamental to something I want to build.