Quantcast

SolrMarc indexer stalling? (was RE: vufind 3.1 indexing error)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SolrMarc indexer stalling? (was RE: vufind 3.1 indexing error)

Delis, Christopher
Has anyone else noticed SolrMarc stall completely during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the mulithreaded processing, but I haven't dug deeper into it. I was just wondering if I was the only one experiencing this. 

I do index very large sets of records (tens of millions) and the stalling tends to occur every couple of million records or so (e.g., indexing 35 million records recently, it occurred 9 or 10 times). The marc record files are at most 1GB each. I've written a monitor script that looks for stalled index jobs and restarts them automatically, but that's just a workaround. Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single marc record file (1GB).

I'm about to try upgrading to the latest SolrMarc in case it helps, but I would love to know if others were experiencing this, too. 

Thanks,
Chris

From: Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 10:22 AM
To: Zhang, Fan
Cc: [hidden email]
Subject: Re: [VuFind-General] vufind 3.1 indexing error

The SolrMarc indexer found in release 3.1 had some bugs. We released an improved version in 3.1.1, and even more fixes will be incorporated in 3.1.2 (due early next year). If you want the latest SolrMarc, you can get it here:

 

https://github.com/solrmarc/solrmarc/releases

 

You should be able to unpack the SolrMarc release into VuFind’s import folder, and rename the solrmarc-core-x.x.x.jar to solrmarc-core.jar to upgrade manually.

 

If you’re still having trouble, please let me know… and I apologize for the inconvenience! SolrMarc 3.x is going to be a big improvement over 2.x both in features and performance, but these types of big transitions usually have some bumps along the way.

 

- Demian

 

From: Zhang, Fan [mailto:[hidden email]]
Sent: Monday, December 19, 2016 11:18 AM
To: Demian Katz
Cc: [hidden email]
Subject: vufind 3.1 indexing error

 

Hi Demian,

 

I am working on upgrading our vufind from 2.3 to 3.1. I downloaded vufind-3.1.tar.gz and tried to digest a couple of marc files. First I got an error showing that there is no lib-solrj in the release tar. I went to solrmarc site and downloaded the master branch zip file, and copied the lib-solrj drectory to vufind/import, now I am getting following error:

 

ERROR [main] (IndexDriver.java:119) - Error processing index configurations: marc.properties, marc_local.properties

ERROR [main] (IndexDriver.java:333) - collection =  9975

ERROR [main] (IndexDriver.java:335) - collection : Error: Illegal character <5>  found in scanner state subfield

ERROR [main] (IndexDriver.java:333) - institution =  9974

ERROR [main] (IndexDriver.java:335) - institution : Error: Illegal character <4>  found in scanner state subfield

 

Please help, thanks!

 

Fan Zhang

Application Developer

PALS, Minnesota State University, Mankato

Mankato, MN 56001

507-389-1975

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Loading...