improve cleaner memory usage

[lopuhin]

don't hold removed elements in memory to reduce memory usage, as they are being merged into parent elements and element text is growing. On some pathalogical HTMLs (a few MB in size) this leads to hundreds of MB of a difference.

[lopuhin]

Thanks @Gallaecio , just before closing, let me see if I can add a standalone repro which demonstrates the improvement.

[lopuhin]

I was able to create a stand-alone repro for a few hundred MB difference but it needs a custom HTML5 parser and a custom HTML file. As for a completely stand-alone repro, the difference is much smaller, here it is:

$ cat ../clear-html/memory_usage.py 
import resource

from lxml.html import fromstring
from clear_html import clean_node, cleaned_node_to_html


def main():
    start_memory_usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    print('memory usage at the start', start_memory_usage)
    for _ in range(3):
        depth = 250
        html = "<article>" + "\n".join(
            (" " * i) + "<div>" + (f"{i} some very long text" * 10000)
            for i in range(depth)
        ) + "</div>" * depth + "\n</article>"
        node = fromstring(html)
        cleaned_node = clean_node(node)
        cleaned_html = cleaned_node_to_html(cleaned_node)
    print('original html len:', len(html), 'cleaned html len:', len(cleaned_html))
    print('html', html[:100])
    print('cleaned html', cleaned_html[:100])
    end_memory_usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    print('memory usage at the end', end_memory_usage)
    print('memory usage growth', end_memory_usage - start_memory_usage)


if __name__ == '__main__':
    main()

Running on main:

$ python ../clear-html/memory_usage.py 
memory usage at the start 20568
original html len: 56434144 cleaned html len: 56402271
html <article><div>0 some very long text0 some very long text0 some very long text0 some very long text0 
cleaned html <article>

<p>0 some very long text0 some very long text0 some very long text0 some very long text0 
memory usage at the end 501676
memory usage growth 481108

Running on this branch:

$ python ../clear-html/memory_usage.py 
memory usage at the start 20584
original html len: 56434144 cleaned html len: 56402271
html <article><div>0 some very long text0 some very long text0 some very long text0 some very long text0 
cleaned html <article>

<p>0 some very long text0 some very long text0 some very long text0 some very long text0 
memory usage at the end 459280
memory usage growth 438696

So the difference here is only around 42 MB, I guess to reproduce a larger difference a more elaborate HTML structure is required.