500 Internal Server Error when fetching Plex Watchlist using RSS

[AhmedNSidd]

It's been reported by multiple users that there's been a new error that's been showing up when using watchlistarr:

2024-06-21 17:32:36.096624+02:0015:32:36.095 [io-compute-0] WARN  PlexTokenSync$ - Unable to fetch watchlist from Plex: org.http4s.client.UnexpectedStatus: unexpected HTTP status: 500 Internal Server Error for request GET https://rss.plex.tv/[[REDACTED]]?format=json

I've also been running into this issue as well. When I open the link that's logged in my browser, I can see the watchlist perfectly fine, but it's not working in watchlistarr..

[nylonee]

Yeah I'm seeing this issue too on Watchlistarr, but when I try to copy the same request I make into Postman I don't get a 500 error... Odd

[nylonee]

Annoyingly after turning on debug logs, the error disappeared and now I can't replicate it... Did it get fixed for you too?

[AhmedNSidd]

@nylonee Same thing happened here. I messed around with some things. I went into my Plex Account settings -> went to watchlist settings -> deleted the active watchlist url -> restarted my docker watchlistarr containers.

After doing that, watchlistarr seemed to start working, and I started to get the 500 Internal errors again after the first poll got the requests successfully.

My working theory right now is that they've started rate limiting the RSS feeds and are sending 500 Internal Errors now for repeated requests to the same Plex Watchlist RSS URLs.

[jamcalli]

I can confirm that the 500 error does not show in the logs. However, watchlistarr is not functioning and nothing is sent to any of the arrs. When I restart the container and it does its initial token sync it works. Afterwords, the rss sync is not adding anything. There is nothing in the logs to provide more insight.

[AhmedNSidd]

Did some relevant experimentations: Deleted the previous "bad" Plex Watchlist RSS URL, restarted the watchlistarr docker containers, and bumped the time interval for fetching the watchlists from 10 seconds -> 1 minute, and then 10 seconds -> 10 minutes.

Both times, the first request was obviously accepted, and the watchlist was fetched, but the second time, I got a 500 Internal Error for both 1 minute and 10 minute intervals.

If Plex did indeed purposefully setup ratelimitting on Watchlist RSS urls, then I don't know what they've set the intervals to. Probably something more than 10 minutes (maybe somewhere between 20 - 60?) Someone else would need to experiment to be able to say for sure, but this change from Plex seems like it pretty much breaks the goal of Watchlistarr, unfortunately.

[nylonee]

Could I get one of you to try this curl command, but with your own plex RSS URL pasted in?

curl --location 'https://rss.plex.tv/YOUR_RSS_ID?format=json' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'User-Agent: watchlistarr/1.0' \
--header 'host: rss.plex.tv'

I wrote a mini script to hit this URL 10 times per second and let it run for 5 minutes, and I didn't get the 500 error even once, so I think it's only happening for some ids? Odd

[AhmedNSidd]

@nylonee The situation is a bit complex. Here's my understanding:

Once a new Plex Watchlist RSS URL is generated for your account, you should be able to get Plex Watchlist items. BUT if you query that URL really frequently, then the first few times you MIGHT see 500 Internal Server Errors, but eventually, you'll start getting no errors at all. But not getting any 500 Internal Server Error is not necessarily a good thing. After this happens, any new watchlist items that you add will not be reflected in the api results that you get now.

You can try this out for yourself too. Try using your miniscript for a Plex Watchlist RSS URL. Then, add a new Plex Watchlist item and use the same URL you were using. I bet you won't see the new Plex Watchlist item show up in your query results.

My understanding is that at first, it will give you 500 Internal Server errors if you try to query the URL too much, but eventually, it stops even trying to fetch new results, and it just automatically returns the cached results to you. My guess would be that eventually, after a certain amount of time once the rate limit has passed, it will stop giving you cached results and give you the actual results, but I'm not sure about this.

[nylonee]

Then, add a new Plex Watchlist item and use the same URL you were using. I bet you won't see the new Plex Watchlist item show up in your query results.

You're onto something here, I added an item to my plex watchlist, it's been a few minutes but it's still not showing up on the RSS feeds? I think Plex has added a caching layer to the request. And this layer is probably 500 error-ing out

It might be worth waiting until the weekday to see if Plex can fix this issue on their side

[AhmedNSidd]

It might be worth waiting until the weekday to see if Plex can fix this issue on their side

Yeah, I think that's a good idea. It could be the case that with the new developmental changes that they pushed (that you noticed), they added this caching mechanism but there's some bugs in it.

[AhmedNSidd]

@nylonee Did a bit more testing.

• Since watchlistarr is for whatever reason not able to poll from the RSS URL, I noticed something else: Watchlistarr is able to successfully fetch any new watchlist items from the Plex Watchlist when the Watchlistarr docker container is first booted up. If I had to guess, this must be because when watchlistarr is first booted up, it fetches Plex Watchlists using a different method than the RSS method, but I haven't looked at the code to confirm this. Anyway -- through this, I was able to find a temporary workaround where I'm restarting the watchlistarr docker container every 20 minutes (using cron) to fetch any new Plex Watchlist items.
• I think Watchlistarr's logic for fetching data from the RSS Plex Watchlist requests is just broken atm. I think this because I was doing some testing and I ran into a scenario where I was able to validate that the RSS Plex Watchlist was successfully updated after I had added a new Plex watchlist item, and I validated this by using curl, but Watchlistarr did not detect these changes, and did not add it to Radarr as expected. There could be two reasons why this happened: either the cached results are still being returned to Watchlistarr, which I don't know why that would be the case since curl showed new changes, or the 2nd option is that they changed the way the data is returned for the Plex Watchlist RSS URL, and Watchlistarr needs to respect the new contract.

EDIT: I looked more into the code for Watchlistarr, and it seems like the models are reasonable, and they should be able to parse the json that the rss feeds are giving, so I'm not sure why curl would be giving me new watchlist items, but these are not detected by watchlistarr itself....

[nylonee]

For your first point, yes you are right, watchlistarr does a full sync on the first run. The Boolean configuration for that is defined here.

I'm thinking as a temporary workaround I can allow the 'token' sync to run every 20 minutes, instead of just on first startup. The token sync is crazy expensive, for me for instance it takes 3 minutes to run and fetch all the data it needs from Plex.

[seb20137]

Bonjour tout le monde j'ai regardé de mon côté, je suis sous unraid avec un container plex et watchlistarr et j'ai bien l'erreur Internal 500, j'ai comparé les logs de plex, l'ajout des films RSS et watchlistarr et j'ai remarqué de mon côté que le flux RSS de plex est en retard de 2 heures par rapport à tout ( je suis a la bonne heure sur mon bios, unraid, plex ... ) Ce qui pourrait expliquer le problème de synchronisation, j'ai pallier le problème en lançant un script qui redémarre mon container Watchlistarr toute les 5 mn et la aucun problème. Et de ce fait pas besoin de plex pass.

Hello everyone, I looked on my side, I am under unraid with a plex and watchlistarr container and I have the Internal 500 error, I compared the plex logs, added the RSS and watchlistarr films and I noticed for my part that the plex RSS feed is 2 hours late compared to everything (I am at the right time on my bios, unraid, plex...) Which could explain the synchronization problem , I solved the problem by launching a script which restarts my Watchlistarr container every 5 minutes and no problem. And therefore no need for a plex pass.

[nylonee]

Thanks @seb20137, I've looked at the progress on the Plex side and it doesn't look like they've done much in the past day.

I'll get to implementing a hotfix so that people don't need to restart Watchlistarr to get a full sync happening.

This will be in a few parts:

1. Do a full plex sync every 20 minutes
2. Suppress the 500 error logs, as there's not much we can do about it
3. (non-urgent) Better logging around the issue so that we know when Plex has resolved the intermittent 500 errors and caching

[nylonee]

Part 1 is done here: #165
Could I get some eyes on the PR?

[nylonee]

Part 2 is done here: #166

[nylonee]

I've bumped the beta version on docker to a version that syncs every 20 minutes, and is less noisy about the 500 errors.

I'll give it a try for a day or so and if there are no issues then I'll make a release

[ersan]

Just started playing with watchlistarr today - there appears to be a fairly long server-side cache set up for RSS feeds on Plex's end, possibly unintentionally. It looks like 10 minutes at least before the cache expires. Meaning the program doesn't pick up new entries added to your watchlist for at least 10 minutes now.

This sort of defeats the purpose of the watchlistarr - I might as well just keep using Overseerr with its 20-minute updates.

As a trick, you can add random parameters to the end of the RSS URL to get it to bypass the cache (ex: ?12345 or ?arg=56789) but I don't know if Plex is going to appreciate that.