imjl

2007-11-05

Lots of good things happening in Lucene land lately, all of which should benefit users with faster indexing and searching capabilities. Most notably, Lucene 2.3 (hopefully released this quarter) has some major changes in indexing memory management and performance. I have personally clocked indexing using release 2.2 at about 400 rec/s (single threaded, Mac Pro dual CPU/dual core, using the contrib/benchmark indexing.alg) to over 2,100 records/s on 2.3-dev (the latest trunk). It also features easier control of the indexing process by specifying how much memory to give it, instead of the confusing maxBufferedDocs factor.

Other work being undertaken should speed up reopening IndexReader’s. There also are a number of smaller changes including a faster StandardTokenizer (the tokenizer most people use) and faster term vector access.

Of course, with that comes more testing and a greater need to make sure the next release is rock solid and backwards compatible. So, if you are a Lucene user, I would encourage you to give trunk a try on some of your non-production indexes, etc. and help us test it out.

link from http://lucene.grantingersoll.com/2007/11/02/lucene-goodness/