Fetch-on-demand iterators (was Re: [CDBI] Make CDBI go fast)

Michael G Schwern schwern at gmail.com
Fri Feb 16 16:19:22 GMT 2007

Edward J. Sabol wrote:
> Yeah, but you're not using Sybase with mod_perl, and I am, so hence my
> nervousness.

I've decided, more because it makes the fetch-on-demand iterator faster and simpler, to split up the iterators so we have one Iterator class which reads everything out up front like it does now and another which reads out on demand.  Both are handed a statement handle by CDBI to unify the interface.  I should be able to make CDBI smart enough to choose the right one for the given database, but if it has to default to the pre-fetch iterator that's fine.  The pre-fetch iterator can be plugged in by the user.

The important change is that CDBI hands the iterator the statement handle and not a list.  That's about all I need.  Unfortunately it will probably break people's existing custom iterators.  I haven't gotten a clear answer back from Tony on if that's allowed, but the Iterator interface is pretty undocumented.  Does anyone write custom iterators and if so how custom are they?  I can preserve most well-behaved subclasses.

> Although you're probably just using it as an extreme example, I don't really
> see much point in optimizing the specific case of CDBI->search(...)->first.

It really a performance problem in the code I'm optimizing.  The iterator patch dropped the loading of one of the more popular pages from 2+ seconds to half a second.

There are other options, including adding limits and improving their overall algorithm, but they all involve more code change to their code which I'm trying to avoid.  And, as mentioned, except for search->first it is not obvious what the limit should be.  Finally, I'd rather push efficiencies down into the framework where possible rather than have the programmer remember to do it.  And if you want to put limits on your searches, you still can!

More information about the ClassDBI mailing list