Caching layer for Boto / Botocore libraries.
This project was started to solve the issue raised here.
My day job requires me to write and use multiple tools and standalone scripts to audit AWS environments. Most of these tools are hacky and written in python which uses boto3 / botocore in the backend to interact with AWS API.
Sometimes, I would have to write a wrapper to combine these scripts to get a custom consolidated report. Since these scripts are standalone, they repeat the same API calls that a previous script would have already called before, leading to redundant API calls, throttling and unnecessary IO wait.
These wrapped tools are sometimes even used as Lambda for automation of certain things. The IO wait times becomes a bottleneck when used in environment such as Lambda where execution is a time-bound activity.
Hence I was looking for a caching layer over boto that can solve reducing these direct redundant calls and the wait times. Thus the birth of botocache.
Botocache caches the response of API calls initiated through boto3 / botocore. Any subsequent redundant call will end up getting the previously cached response from Botocache as long as the call is within the expiry timeout of the cached response.
Botocache can work with any cache library that is based on cachetools
It uses the unittest module's patch as the magic component to achieve this. 😉
This project is little hacky given the nature how it achieves caching, but it gets the job done.
Stable release installation (PyPI) :-
pip3 install botocacheTest release installation (Test PyPI)
pip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple botocacheInstallation directly from this Repository:-
pip3 install git+https://github.com/rams3sh/botocache.gitBelow snippet demonstrates usage of botocache.
from boto3.session import Session
from cachetools_ext.fs import FSLRUCache
from botocache.botocache import botocache_context
cache = FSLRUCache(ttl=900, path=".cache", maxsize=1000)
# action_regex_to_cache parameter consists list of regex to be matched against a given action for considering the call to be cached
with botocache_context(cache=cache,
action_regex_to_cache=["List.*", "Get.*", "Describe.*"],
call_log=True, # This helps in logging all calls made to AWS. Useful while debugging. Default value is False.
supress_warning_message=False # This supresses warning messages encountered while caching. Default value is False.
):
cached_session = Session()
# Initialising client with botocache context
cached_client = cached_session.client('iam')
"""
Using cached client for paginating list users
"""
paginator = cached_client.get_paginator('list_users')
for page in paginator.paginate():
print(page)
"""
Always use a fresh initialised session client and subsequent new objects outside the context of botocache to use
boto3 without caching layer.
"""
non_cached_session = Session()
non_cached_client = non_cached_session.client('iam')
paginator = non_cached_client.get_paginator('list_users')
for page in paginator.paginate():
print(page)Note:
Botocache has been tested with cachetools_ext's FSLRU cache, but it should logically work with any cache that is compatible with cachetools.
-
This project was created mainly to support my specific internal use cases. Hence, there is a good scope of it having bugs and functional issues. Feel free to raise a PR / Issue in those cases.
-
Botocache does not understand HTTP related caching specification. It works based on function caching.Its a very simple dumb caching layer that checks for specific attributes in an API call, converts into a key and stores the response against the key. Any subsequent call having matching attributes will be returned with the value stored against the same key.