excessive commits

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

excessive commits

Barnett, Jeffrey
We are trying to get our full set of 7 million records loaded, and along with problems with the records themselves, it seems to take longer and longer to add records.

One thin I notice in the logs is that a  possible excessive amount of committing and deduping on very small record sets.
1) is this normal?
2) is there a control to minimize it?
3) is this a good thing, and I just don't realize why?

Sample Log:
Jul 6, 2008 12:08:54 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Jul 6, 2008 12:08:54 PM org.apache.solr.update.DirectUpdateHandler2 doDeletions
INFO: DirectUpdateHandler2 deleting and removing dups for 89 ids
Jul 6, 2008 12:08:54 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@33f652 DirectUpdateHandler2
Jul 6, 2008 12:10:15 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Jul 6, 2008 12:10:15 PM org.apache.solr.update.DirectUpdateHandler2 doDeletions
INFO: DirectUpdateHandler2 deleting and removing dups for 91 ids
Jul 6, 2008 12:10:15 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@81f8ea DirectUpdateHandler2
Jul 6, 2008 12:10:17 PM org.apache.solr.update.DirectUpdateHandler2 doDeletions
INFO: DirectUpdateHandler2 docs deleted=0
Jul 6, 2008 12:10:17 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@fcf912 main

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech