libmc is a memcached client library for Python without any other dependencies at runtime. It's mainly written in C++ and Cython and can be considered a drop in replacement for libmemcached and python-libmemcached.
libmc is developed and maintained by Douban Inc. Currently, it is working in a production environment, powering all web traffic on douban.com. [1]
| [1] | The link not an endorsement of the traffic estimate site's accuracy, just a publicly available reference point |
For users:
pip install libmc
Usage:
import libmc
mc = libmc.Client(['localhost:11211', 'localhost:11212'])
mc.set('foo', 'bar')
assert mc.get('foo') == 'bar'Under the hood, libmc consists of 2 parts: an internal, fully-functional
memcached client implementation in C++ and a Cython wrapper around that
implementation. Dynamic memory allocation and memory-copy are slow, so
we've tried our best to avoid them. libmc also supports the set_multi
command, which is not natively supported by the memcached
protocol.
Some techniques have been applied to make set_multi command extremely fast
in libmc (compared to similiar libraries).
import libmc
from libmc import (
MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT
)
mc = libmc.Client(
[
'localhost:11211',
'localhost:11212',
'remote_host',
'remote_host mc.mike',
'remote_host:11213 mc.oscar'
],
do_split=True,
comp_threshold=0,
noreply=False,
prefix=None,
hash_fn=MC_HASH_MD5,
failover=False
)
mc.config(MC_POLL_TIMEOUT, 100) # 100 ms
mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms
mc.config(MC_RETRY_TIMEOUT, 5) # 5 sservers: a list of memcached server addresses. Each address should be formated ashostname[:port] [alias], whereportandaliasare optional. Ifportis not given, the default port11211will be used. If given,aliaswill be used to compute the server hash, which would otherwise be computed based onhostandport(i.e. whichever portion is given).do_split: splits large values (up to 10MB) into chunks (<1MB). The memcached server implementation will not store items larger than 1MB, however in some environments it is beneficial to shard up to 10MB of data. Attempts to store more than that are ignored. Default:True.comp_threshold: compresses large values using zlib. Ifbuffer length > comp_threshold > 0(in bytes), the buffer will be compressed. Ifcomp_threshold == 0, the string buffer will never be compressed. Default:0noreply: controls memcached'snoreplyfeature. Default:Falseprefix: The key prefix. default:''hash_fn: hashing function for keys. possible values:MC_HASH_MD5MC_HASH_FNV1_32MC_HASH_FNV1A_32MC_HASH_CRC_32
default:
MC_HASH_MD5NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are per each spec, but they're not compatible with corresponding implementions in libmemcached.
failover: Whether to failover to next server when current server is not available. Default:FalseMC_POLL_TIMEOUTTimeout parameter used during set/get procedure. Default:300msMC_CONNECT_TIMEOUTTimeout parameter used when connecting to memcached server in the initial phase. Default:100msMC_RETRY_TIMEOUTWhen a server is not available due to server-end error, libmc will try to establish the broken connection in everyMC_RETRY_TIMEOUTs until the connection is back to live. Default:5s
NOTE: The hashing algorithm for host mapping on continuum is always md5.
Feel free to send a Pull Request. For feature requests or any questions, please open an Issue.
For SECURITY DISCLOSURE, please disclose the information responsibly by sending an email to [email protected] directly instead of creating a GitHub issue.
No, but, if you like, you can write a wrapper for PHP based on the C++ implementation.
No. Only Memcached ASCII protocol is supported currently.
Before libmc, we were using python-libmemcached, which is a python extention for libmemcached. libmemcached is quite weird and buggy. After nearly one decade, there're still some unsolved bugs.
Yes. libmc.ThreadedClient is a thread-safe client implementation. To hold
access for more than one request, libmc.ClientPool can be used with Python
with statements. libmc.Client, however, is a single-threaded memcached
client. If you initialize a standard client in one thread but reuse that in
another thread, a Python ThreadUnsafe Exception will be raised.
Yes, with the help of greenify,
libmc is friendly to gevent. Read tests/shabby/gevent_issue.py for
details. libmc.ThreadedClient and libmc.ClientPool are not compatible.
[2]
| [2] | In order to use a single executable for multiple greenlet contexts, gevent has to copy thread memory to and from the same stack space. This doesn't affect Python references, which are handed off through gevent, but makes it impossible for shared libraries to pass memory addresses across greenlets, which is required for the worker pool. |
Notice:
gevent.monkey.patch_all() will override
threading.current_thread().ident to Greenlet's ID,
this will cause libmc to throw a ThreadUnSafe error
or run into dead lock, you should only patch the things
that you need, e.g.
from gevent import monkey
monkey.patch_socket()- Thanks to @fahrenheit2539 and the llvm project for the standalone. SmallVector implementation.
- Thanks to @miloyip for the high performance i64toa implementation.
- Thanks to Ivan Novikov for the research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS.
- Thanks to the PolarSSL project for the md5 implementation.
- Thanks to @lericson for the benchmark script in pylibmc.
- Thanks to the libmemcached project and some other projects possibly not mentioned here.
- 豆瓣
- 下厨房
- Some other projects on GitHub
- Want to add your company/organization name here? Please feel free to send a PR!
https://github.com/douban/libmc/wiki
Copyright (c) 2014-2020, Douban Inc. All rights reserved.
Licensed under a BSD license: https://github.com/douban/libmc/blob/master/LICENSE.txt