AiFail

Python Library for retrying openai/gcp API calls

Installation

pip install aifail

Usage

To get started, simply wrap your functions with @retry_if, specifying the condition to retry on.

aifail/examples/basic_openai.py

Line 5 in a540c05

from aifail import retry_if, openai_should_retry

aifail/examples/basic_openai.py

Lines 13 to 30 in a540c05

	@retry_if(openai_should_retry)
	def gpt4(messages):
	response = client.chat.completions.create(model="gpt-4", messages=messages)
	print(response.choices[0].message.content)
	gpt4(
	messages=[
	{
	"role": "system",
	"content": "You are a helpful assistant designed to output JSON.",
	},
	{
	"role": "user",
	"content": "Who won the world series in 2020?",
	},
	]
	)

Custom logic

You can use this with anything that needs retrying, e.g. google sheets -

def sheets_api_should_retry(e: Exception) -> bool:
    return isinstance(e, HttpError) and (
        e.resp.status in (408, 429) or e.resp.status > 500
    )


@retry_if(sheets_api_should_retry)
def update_cell(spreadsheet_id: str, row: int, col: int, value: str):
    get_spreadsheet_service().values().update(
        spreadsheetId=spreadsheet_id,
        range=f"{col_i2a(col)}{row}:{col_i2a(col)}{row}",
        body={"values": [[value]]},
        valueInputOption="RAW",
    ).execute()

Advanced Usage

This library is used by GooeyAI in production to handle thousands of API calls every day. To save costs and handle rate limits, you can intelligently specify quick fallbacks (eg azure openai) -

aifail/examples/azure_openai_fallback.py

Lines 7 to 16 in a540c05

	azure_client = openai.AzureOpenAI(
	api_key=os.environ["AZURE_OPENAI_KEY"],
	azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
	api_version="2023-10-01-preview",
	max_retries=0, # disables openai's internal retry logic
	)
	openai_client = openai.OpenAI(
	api_key=os.environ["OPENAI_API_KEY"],
	max_retries=0, # disables openai's internal retry logic
	)

aifail/examples/azure_openai_fallback.py

Lines 19 to 35 in a540c05

	@retry_if(openai_should_retry)
	def chad_gpt4(messages, max_tokens=4000):
	response = try_all(
	# first try with azure
	lambda: azure_client.chat.completions.create(
	model="openai-gpt-4-prod-ca-1", # replace with your deployment name
	messages=messages,
	max_tokens=max_tokens,
	),
	# if that fails (for whatever reason), try with openai
	lambda: openai_client.chat.completions.create(
	model="gpt-4",
	messages=messages,
	max_tokens=max_tokens,
	),
	)
	print(response.choices[0].message.content)

Traceable Errors

AiFail comes with a built in logger, and outputs complete stack traces tracing the error back to the original call site.

# python examples/azure_openai_fallback.py 
2023-11-18 04:36:01.364 | WARNING  | aifail.aifail:try_all:63 - [2/2] tyring next fn, prev_exc=NotFoundError("Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}")
2023-11-18 04:36:01.778 | WARNING  | aifail.aifail:wrapper:98 - [1/1] captured error, retry_delay=0.4117675457681431s, exc=NotFoundError("Error code: 404 - {'error': {'message': 'The model `gpt-4-x` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}")
2023-11-18 04:36:02.483 | WARNING  | aifail.aifail:try_all:63 - [2/2] tyring next fn, prev_exc=NotFoundError("Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}")
2023-11-18 04:36:04.093 | WARNING  | aifail.aifail:wrapper:98 - [2/1] captured error, retry_delay=0.9974197744911488s, exc=NotFoundError("Error code: 404 - {'error': {'message': 'The model `gpt-4-x` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}")
Traceback (most recent call last):
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 65, in try_all
    return fn()
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 28, in <lambda>
    lambda: azure_client.chat.completions.create(
  ...
openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 86, in wrapper
    return fn(*args, **kwargs)
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 26, in chad_gpt4
    response = try_all(
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 69, in try_all
    raise prev_exc
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 65, in try_all
    return fn()
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 34, in <lambda>
    lambda: openai_client.chat.completions.create(
  ...
openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `gpt-4-x` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 65, in try_all
    return fn()
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 28, in <lambda>
    lambda: azure_client.chat.completions.create(
  ...
openai.NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 43, in <module>
    chad_gpt4(
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 102, in wrapper
    raise prev_exc
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 86, in wrapper
    return fn(*args, **kwargs)
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 26, in chad_gpt4
    response = try_all(
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 69, in try_all
    raise prev_exc
  File "/Users/dev/Projects/dara/aifail/aifail/aifail.py", line 65, in try_all
    return fn()
  File "/Users/dev/Projects/dara/aifail/examples/azure_openai_fallback.py", line 34, in <lambda>
    lambda: openai_client.chat.completions.create(
  ...
openai.NotFoundError: Error code: 404 - {'error': {'message': 'The model `gpt-4-x` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

Process finished with exit code 1

Sentry too, will capture the entire retry loop