[CDBI] Fast fetch-on-demand iterator complete

Michael G Schwern schwern at pobox.com
Thu Feb 22 02:28:33 GMT 2007

I've completed work on the fetch-on-demand iterator.  You can get the patch here:

It requires an Ima::DBI patch to allow finer control of the statement handle cache.

There are now three iterators.
* PreFetch - The old CDBI::Iterator which fetches everything up front
* OnDemand - The new fetch-on-demand CDBI::Iterator
* OnDemand::Opportunistic - More on this below

All the iterators now execute the statement on demand rather than on creation.  This means the iterators do not contain an active statement handle until they're used.  You can change this in an iterator subclass if you like.

PreFetch beats OnDemand by about 10-20% if all objects are retrieved.  OnDemand beats PreFetch if only a few are retrieved.  OnDemand will cache the rows fetched so it can efficiently support things like reset() and first().  If OnDemand has to fetch out all of its rows, such as with count(), it will simply become a PreFetch iterator.

OnDemand::Opportunistic is a hybrid.  It starts life as an OnDemand iterator.  After 100 fetches (configurable) it becomes a PreFetch iterator.  This gives it the high performance of an OnDemand iterator if the whole list is not iterated through and the high performance of the PreFetch if the whole list is used.  Its a good candidate to be the default CDBI iterator.

There are incompatibilities:
* CDBI::Iterator is now abstract.  The code has been moved to CDBI::Iterator::PreFetch.
* CDBI::Iterator->new() has changed to take a hash of parameters.

Anyone who subclasses from CDBI::Iterator or uses it directly now won't work.
I've done a google code search and a number of modules will be effected.

* Delivery::Machinery::Iterator
* CDBI::Factory
* CDBI::Plugin::Iterator
* Handel::Iterator
* Buscador::Addressing

As the number is small, the iterator interface was never really documented anyway and the new interface is much more flexible I think it is worth it.  The authors can be contacted.

More information about the ClassDBI mailing list