Handle Anthropic overloaded errors during streaming

[doomspork]

As we're making more use of streaming we're seeing more of these errors recently:

[error] Unsupported event received when parsing Anthropic response: "event: error\ndata: {\"type\":\"error\",\"error\":{\"details\":null,\"type\":\"overloaded_error\",\"message\":\"Overloaded\"}        }"

The result is this error logged but the code seems to proceed like everything is okay. I would expect/prefer an error were raised so we could retry the conversation versus leaving it in a partial and incomplete state.

I started spelunking today but it looks like handling this could be a decent sized changes so I thought it was best to bring it here for discussion.

Anthropic documentation around streaming errors for reference: https://docs.anthropic.com/en/api/messages-streaming#error-events

[brainlid]

Just saw this myself this morning! Thanks for reporting it.

[brainlid]

There's more to handling this than I've done so far. It helps that they keep making it reproducible. 🤦‍♂️

[doomspork]

I haven't had a chance to pull your changes and do my own testing but I will here shortly.

It's unfortunate how frequently their API is overloaded recently.

[brainlid]

Yeah. I’m going to take a different approach. I’ll try raising an exception and catching it in the LLMChain. So no, it’s not working great yet.

…

On Fri, Nov 22, 2024 at 7:37 AM Sean Callan ***@***.***> wrote: I haven't had a chance to pull your changes and do my own testing but I will here shortly. It's unfortunate how frequently their API is overloaded recently. — Reply to this email directly, view it on GitHub <#193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGFQGGHCD4C2EKJRPPBGJT2B46TPAVCNFSM6AAAAABSHHUKYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJTHEYDSMBWHA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[brainlid]

@doomspork I pushed up a hopefully more complete fix.

When it happens again, the LLMChain.run(chain, ...) result will be: {:error, updated_chain, %LangChain.LangChainError{} = error}.

Where the error has a type of "overloaded_error" (that's what they return).

If the pattern holds, it will be down again tomorrow morning (but it's the weekend, so perhaps not). I'll be watching for it too though.

[doomspork]

Sounds good @brainlid! Thanks for tackling this one 🙏 I'll let you know what I see 🫡