more memory savings #222

Villemoes · 2017-02-10T13:26:29Z

No description provided.

The values we store in the .expand_caches are immutable (if we make a change to one of the vars in the dependency set, the entry gets deleted). This means we may share those tuples among all the metadata instances, saving a significant amount of memory. First, this requires changing the second member to a frozenset, and that reveals a (harmless) bug in the override handling; one shouldn't mutate the dependency set one gets from a _get call (the effect was just to add "OVERRIDES" to the dependency set for the "OVERRIDES" variable itself). If we have other such bugs hiding somewhere, the change to using a frozenset should flush them out as well. Then it's just a matter of creating two dicts for holding the interned frozensets and tuples, using the same strategy as Python uses internally for interning strings. As a bonus, we get to drop the optimization where we used None instead of an empty set, since we now use the singleton empty frozenset. The saving is around 7% (40M for oe bake world), with no measurable effect on initialization time. I've done the usual dump of the metadata before and after to check that this does not affect hashes. The dict holding the interned tuples ends up growing to a few MB, so clear the caches after we've done all the hash computations. That may or may not reduce our memory footprint by those MB, depending on the details of Python's and glibc's memory allocation strategy.

I was wondering how we ended up creating ~5000 copies of the strings "machine" and "arm-926ejs-linux-gnueabi" (each thus costing about 3M of memory). It turns out these are the .type and .arch members of the OElitePackage instances, and the reason they each have their own copy is that the string objects are fresh out of the sqlite database. Let's ask pysqlite to intern every string it passes back to us. This saves a lot of memory, and is virtually free.

Every time we do the '(provides or "").split()', python obviously creates a new set of string objects. We stash the strings returned in the __provides flag for a lot of DEPENDS_* and RDEPENDS_* variables, which means we end up having thousands of copies of strings like "libc-6", "libpthread" etc. etc. Interning the strings before passing them back ends up saving more than 50000 string objects, or about 3M of memory.

We cache the compiled actual python functions in the global pythonfunc_code_cache; the PythonFunction wrappers are not and AFAICT cannot be cached (since they have mutable state preventing reuse). So delete the unused attribute, saving about 1M for an oe bake world (280 bytes * 4500 MetaData instances).

The memory footprint of an ExpansionStack instance is 72 bytes for the instance itself, 280 bytes for the __dict__ attribute, and then the memory for the actual list stored at the .stack attribute. We can get rid of the first two contributions by simply letting an ExpansionStack be a list with one extra method that does the sanity checking. Conveniently, the builtin list type even already has a .pop method which does the right thing. Also, __init__ and __len__ are already taken care of. Nobody calls __str__ directly allowing them to pass the optional argument and str(foo, "something else") would be a runtime error, so just make __str__ have the usual prototype. The .python attribute never seems to have been used for anything. Saving 352 bytes per instance adds up to about 1.5M for an oe bake world.

Rasmus Villemoes added 5 commits February 9, 2017 10:57

esben added the in progress label Feb 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

more memory savings #222

more memory savings #222

Villemoes commented Feb 10, 2017

more memory savings #222

Are you sure you want to change the base?

more memory savings #222

Conversation

Villemoes commented Feb 10, 2017