Fetch-on-demand iterators (was Re: [CDBI] Make CDBI go fast)
Michael G Schwern
schwern at gmail.com
Thu Feb 15 17:12:37 GMT 2007
Perrin Harkins wrote:
> On 2/15/07, Michael G Schwern <schwern at gmail.com> wrote:
>> Whatever the backend implementation, it sure sped things up. The
>> performance of search()->first when the search returns a million rows
>> went from about 2 seconds to almost negligable. Remember, even if the
>> database is doing the same amount of work it still has to transfer it
>> to your process and then run it through DBI and then bind it all into
>> a list of hashes.
> With DBD::mysql (and others), it will still transfer the rows to your
> process, even if you never fetch them. Using "mysql_use_result"
> prevents this.
Looks handy and dangerous. Thanks.
>> Do not be fooled! There really is no "give me everything" bulk row
>> loading optimization in DBI. selectall/fetchall is just doing
>> $sth->fetch in a loop!
> It does it in a fast internal C loop though, which does help.
You're right, I keep getting fooled by those pure Perl implementations.
More information about the ClassDBI