[Chicago-talk] search indexers

[Andrew Brosnan]

On 1/20/06 at 4:29 PM, jt at plainblack.com (JT Smith) wrote:

> > I maintain some code that does this also. I found that as table
> > size and load increase, complex queries can really hang. We're
> > waiting for MySQL Cluster to support FULLTEXT before doing a
> > rewrite. <shrug>
> 
> At what point do you see this, and on why kind of hardware? Are we
> talking 100k rows, 1M, 1b? And what's the average size of your
> records? Or perhaps this is more easily expressed as, what is the
> size of your database table on the file system?

This is two dedicated load balanced db servers (master/slave
replication) each with several GB of RAM. Average MyISAM table sizes
range 1-2 million records (200-400MB). We've spent some time tuning
MySQL and optimizing queries, but the FULLTEXT queries can still cause
problems especially under heavier loads (500K+ per day).

The point I wanted to make (but neglected to) was this - I can pound
away on this in a dev environment using 100K records and everything is
peachy. It's not until we put it in production and things get loaded up
that we see problems. I learned that my 'testing' wasn't really telling
me everything I wanted to know :-)

Regards,
Andrew