[question] Log is flooded with entries about missing accounts

[Karmalakas]

I have my GTS instance running with 2 accounts. One of these accounts has a follower from mastodon.social and I noticed the log is flooded with entries like:

2024-11-14 08:46:31.556434+00:00timestamp="14/11/2024 10:46:31.556" func=server.init.func1.Logger.13.1 level=ERROR latency="113.144518ms" userAgent="Mastodon/4.4.0-nightly.2024-11-13 (http.rb/5.2.0; +https://mastodon.social/)" method=POST statusCode=500 path=/users/trakttv/inbox clientIP=188.245.228.4 pubKeyID=https://mastodon.social/users/Cykris#main-key errors="Error #01: AuthenticateFederatedRequest: error dereferencing account https://mastodon.social/users/Cykris: enrichAccount: error webfingering remote account [email protected]: fingerRemoteAccount: error webfingering @[email protected]: account has been deleted/is gone\n" requestID=2d4dmack04001wxk5100 msg="Internal Server Error: wrote 54B"

and

2024-11-14 08:49:38.296005+00:00timestamp="14/11/2024 10:49:38.295" func=dereferencing.(*Dereferencer).GetAccountByURI.func1 level=ERROR msg="error fetching account featured collection: dereferenceCollection: error dereferencing https://mastodon.social/users/ARBIP/collections/featured: Dereference: GET request to https://mastodon.social/users/ARBIP/collections/featured failed: status=\"410 Gone\" body=\"{\"error\":\"Gone\"}\""
2024-11-14 08:49:38.331765+00:00timestamp="14/11/2024 10:49:38.331" func=dereferencing.(*Dereferencer).dereferenceAccountStats level=WARN msg="couldn't count followers for @[email protected]: dereferenceCollection: error dereferencing https://mastodon.social/users/ARBIP/followers: Dereference: GET request to https://mastodon.social/users/ARBIP/followers failed: status=\"410 Gone\" body=\"{\"error\":\"Gone\"}\""
2024-11-14 08:49:38.367915+00:00timestamp="14/11/2024 10:49:38.367" func=dereferencing.(*Dereferencer).dereferenceAccountStats level=WARN msg="couldn't count following for @[email protected]: dereferenceCollection: error dereferencing https://mastodon.social/users/ARBIP/following: Dereference: GET request to https://mastodon.social/users/ARBIP/following failed: status=\"410 Gone\" body=\"{\"error\":\"Gone\"}\""
2024-11-14 08:49:38.412394+00:00timestamp="14/11/2024 10:49:38.412" func=dereferencing.(*Dereferencer).dereferenceAccountStats level=WARN msg="couldn't count statuses for @[email protected]: dereferenceCollection: error dereferencing https://mastodon.social/users/ARBIP/outbox: Dereference: GET request to https://mastodon.social/users/ARBIP/outbox failed: status=\"410 Gone\" body=\"{\"error\":\"Gone\"}\""

Some account names repeat constantly. When I try to check these accounts, Mastodon just opens a 404 page (which makes sense I suppose)

Where do these accounts come from? Why would GTS keep checking for them over and over if they are gone?

[NyaaaWhatsUpDoc]

this usually ends up happening when mastodon sends out a delete account request. they send out the initial delete, then individual deletes for every single item created by that account that might live on remote servers. but in our case, as soon as we receive the delete account request we just immediately delete everything, so all the subsequent requests fail. it's probably something we can handle a little better, and honestly it would be much more efficient if mastodon performed similarly to us which seems like the more sensible way to sit :p

[tsmethurst]

Yeah I think we could maybe put these particular logs at a debug level rather than warn or error, as they can safely be ignored.

[Karmalakas]

Should this be logged at ERROR level then? I mean I'd understand this for local accounts, but now... I mean, if remote account is already gone on that remote instance, why would I care about that? I'd expect GTS to handle this in the background and log at DEBUG level maybe 🤔 Now these accounts were never related to my instance at all, hence my confusion 😕

[tsmethurst]

Just to add a bit of background/explanation to this, mastodon.social and other very large, open sign-up instances are particularly prone to sending out lots of Delete messages as they undergo quite a lot of account churn -- people sign up for spam accounts and get suspended, or people sign up, change their mind, and delete their account. When that happens, mastodon.social sends out Deletes to everyone the server knows about, repeatedly. And mastodon.social knows about a lot of other instances :')

Unfortunately, Mastodon signs those Delete messages with the public key of the deleted account (good, correct), but it doesn't keep serving the public key or webfinger response for those deleted accounts (questionable), so if your instance hasn't seen the account yet and cached the public key locally, it's impossible to validate the authenticity of those Delete messages.

Actually we should just throw away Delete messages whose key we can't retrieve + validate, but we'd have to do some rearchitecting to do that, as validating public/private key signatures is an integral part of how GoToSocial works.