support lazy batching again, support general iterators

[d-maurer]

Fixes #75.

dtml-in's lazy batching was broken by listifying the incoming sequence in order to support some iterators (e.g. dict key views).
This PR replaces the listifying by a sequence_ensure_subscription wrapper ensuring that the returned object has sequence like behavior sufficient for dtml-in. It uses the original object if this already fulfills the requirement, otherwise a wrapper which lazily emulates sequence behavior for an arbitrary iterator.

The main problem with the approach is the recognition whether the incoming sequence already fulfills the requirements. For this it uses the following heuristics:

• it accesses seq[0]. If this raises IndexError it assumes "sufficiently sequence like"; if it raises AttributeError, TypeError or KeyError it assumes "not sufficiently sequence like"
• it tries to verify that seq is not a mapping. For this it accesses seq[None]. Most sequence types will raise an exception if the index is not an integer or a slice. The heuristics checks for TypeError, ValueError and RuntimeError (the latter because ZTUtils.Lazy.Lazy used this exception. If one of those exceptions is raised, it assumes "sufficuetly sequence like".

The approach is not very robust as it makes assumptions about the raised exceptions. Nevertheless, the PR relies on explicit exception enumeration because in the context of Zope temporary exceptions can arise which we do not want to catch. I am open for other heuristics.

[dataflake]

Looks good to me but I don't have any good test case to see what difference it makes.

[d-maurer]

but I don't have any good test case to see what difference it makes.

I hope that @sauzher will check this with his large catalog.

[sauzher]

@d-maurer
I've checked with ~50.000 objects populated catalog.

It still takes ~7000ms to load. The call that takes time is

this inner call

[d-maurer]

Alessandro Ceglie wrote at 2024-9-30 09:39 -0700:

@d-maurer I've checked with ~50.000 objects populated catalog. It still takes ~7000ms to load. The call that takes time is [this one](https://github.com/zopefoundation/DocumentTemplate/blob/d67f09646acf7cc7d4a1d75bc1e3fbd1b824cee7/src/DocumentTemplate/DT_In.py#L463)

This is surprising: according to your finding, the expensive line is `sequence = sequence_ensure_subscription(md[name])`; the `md[name]` is a simple lookup (very fast); the only expensive thing inside `sequence_ensure_subscription` is `sequence[0]`, i.e. the access to the first search result element. The time for this access should not depend on the search result size, i.e. it should be necessary even if a filter has been applied. Please dig in further. I suggest to profile in an interactive Python interpreter the call `sequence_ensure_subscription(catalog.searchAll())`. The `searchAll` should be very fast (as the operation returns a lazy result). I expect that almost all time is spend in `Lazy.__getitem__` (implementing the `sequence[0]`). Look what for time is spent in this call. I expect ZODB reads; if this is true then ZODB caching (and below maybe OS caching) will highly affect the timing.