Quantcast

SolrMarc Indexer Hanging? Deadlock issue?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SolrMarc Indexer Hanging? Deadlock issue?

Delis, Christopher
[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Demian Katz
I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.

I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.

- Demian

-----Original Message-----
From: Delis, Christopher [mailto:[hidden email]]
Sent: Monday, December 19, 2016 4:30 PM
To: [hidden email]
Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel _______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C7de836f1269d40d1958208d428593b02%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=ysxtgzO4X2ocu%2FSBiurQvRrJo%2B8Nr0ypqPPTL%2Fv3MkM%3D&reserved=0

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Robert Haschart
One thing that might be useful (if it works) is when the program hangs, do a CRTL-\   (ctrl backslash) or send a QUIT signal  to the process:

kill -3 <pid>

It should cause the program to dump info on all of the threads in the program.

-Bob Haschart
________________________________________
From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 5:00 PM
To: Delis, Christopher; [hidden email]
Cc: [hidden email]
Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?

I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.

I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.

- Demian

-----Original Message-----
From: Delis, Christopher [mailto:[hidden email]]
Sent: Monday, December 19, 2016 4:30 PM
To: [hidden email]
Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel _______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C7de836f1269d40d1958208d428593b02%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=ysxtgzO4X2ocu%2FSBiurQvRrJo%2B8Nr0ypqPPTL%2Fv3MkM%3D&reserved=0

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/solrmarc-tech.
For more options, visit https://groups.google.com/d/optout.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Oliver Obenland
We encountered the same problem. We used jstack to get some informations
about the threads. I attached the output. We are using v3.0.2, so the
lines have changed slightly, but you can see that all threads are
sleeping and waiting for something.

We think the problem is between line 83 and 95 of
org.solrmarc.driver.IndexerWorker. We replaced it with

try {
     docQ.put(recDoc);
}
catch (InterruptedException ie)
{
     Thread.currentThread().interrupt();
}

This helped us so far but we still are testing. I think the whole
synchronization stuff should be handled by those BlockingQueues, so the
synchronize-wait-notify-think can be avoided. Tomorrow I'll try to
cleanup our changes and create a pull request.

- Oliver

Am 20.12.2016 um 01:49 schrieb Haschart, Robert J. (rh9ec):

> One thing that might be useful (if it works) is when the program hangs, do a CRTL-\   (ctrl backslash) or send a QUIT signal  to the process:
>
> kill -3 <pid>
>
> It should cause the program to dump info on all of the threads in the program.
>
> -Bob Haschart
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
> Sent: Monday, December 19, 2016 5:00 PM
> To: Delis, Christopher; [hidden email]
> Cc: [hidden email]
> Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?
>
> I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.
>
> I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.
>
> - Demian
>
> -----Original Message-----
> From: Delis, Christopher [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 4:30 PM
> To: [hidden email]
> Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?
>
> [Reposting this from vufind-general because it seems more appropriate here...]
>
> Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.
>
> I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.
>
> Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).
>
> I've upgraded to the latest SolrMarc, but still experience the same hanging issue.
>
> Thanks,
> Chris
>
> P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel _______________________________________________
> Vufind-tech mailing list
> [hidden email]
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C7de836f1269d40d1958208d428593b02%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=ysxtgzO4X2ocu%2FSBiurQvRrJo%2B8Nr0ypqPPTL%2Fv3MkM%3D&reserved=0
>
> --
> You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To post to this group, send email to [hidden email].
> Visit this group at https://groups.google.com/group/solrmarc-tech.
> For more options, visit https://groups.google.com/d/optout.
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
> _______________________________________________
> Vufind-tech mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/vufind-tech

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech

jstack.txt (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Bill Dueber-2
In reply to this post by Delis, Christopher
Are you sure it's hanging, as opposed to just waiting for your solr to perform a merge? The fact that it's periodic makes me wonder if it's the latter.

On Mon, Dec 19, 2016 at 4:29 PM, Delis, Christopher <[hidden email]> wrote:
[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech



--
Bill Dueber
Library Systems Programmer
University of Michigan Library

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Delis, Christopher
Yes, the solr server is sitting idle at the time. Also, it's not periodic; it's random. When it does hang, it hangs for (presumably) ever, until I kill it. The "hung" process will periodically awake (uses small amount of CPU) and then go right back to sleep. --Chris



From: [hidden email] [[hidden email]] on behalf of Bill Dueber [[hidden email]]
Sent: Tuesday, December 20, 2016 1:10 PM
To: Delis, Christopher
Cc: [hidden email]
Subject: Re: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

Are you sure it's hanging, as opposed to just waiting for your solr to perform a merge? The fact that it's periodic makes me wonder if it's the latter.

On Mon, Dec 19, 2016 at 4:29 PM, Delis, Christopher <[hidden email]> wrote:
[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech



--
Bill Dueber
Library Systems Programmer
University of Michigan Library

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Delis, Christopher
In reply to this post by Robert Haschart
In case it helps any, here are the results of the kill -3 <PID>:

2016-12-20 15:42:07
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):

"SolrUpdate-6162166-6162823" #23 prio=5 os_prio=0 tid=0x00007fa7885bb000 nid=0xf156 waiting on condition [0x00007fa76c9e3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6162824-6163477" #22 prio=5 os_prio=0 tid=0x00007fa7885ba000 nid=0xf155 waiting on condition [0x00007fa76d426000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6164123-6164771" #21 prio=5 os_prio=0 tid=0x00007fa7885b9000 nid=0xf154 waiting on condition [0x00007fa76d628000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6163478-6164122" #20 prio=5 os_prio=0 tid=0x00007fa7885b3000 nid=0xf153 waiting on condition [0x00007fa76d527000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"RecordIndexer-Thread-0" #19 prio=5 os_prio=0 tid=0x00007fa7885b3800 nid=0xf152 in Object.wait() [0x00007fa76d729000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.solrmarc.driver.IndexerWorker.run(IndexerWorker.java:82)
        - locked <0x00000006c03f5590> (a java.util.concurrent.ArrayBlockingQueue)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"MarcReader-Thread" #18 prio=5 os_prio=0 tid=0x00007fa7885b1800 nid=0xf151 waiting on condition [0x00007fa76d123000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.MarcReaderThread.run(MarcReaderThread.java:40)

"Eclipse-Shutdown-Simulator-Thread" #16 prio=5 os_prio=0 tid=0x00007fa7885ae000 nid=0xf150 waiting on condition [0x00007fa76d224000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.IndexDriver$ShutdownSimulator.run(IndexDriver.java:536)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fa788136800 nid=0xf146 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa788133800 nid=0xf145 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa788130800 nid=0xf144 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa78812e800 nid=0xf143 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa788107000 nid=0xf142 in Object.wait() [0x00007fa76e79c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa788102800 nid=0xf141 in Object.wait() [0x00007fa76e89d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
        at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:295)
        at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:100)
        at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:113)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.Boot.main(Boot.java:72)

"VM Thread" os_prio=0 tid=0x00007fa7880fa800 nid=0xf140 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa78801f800 nid=0xf13e runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa788021000 nid=0xf13f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fa78813d000 nid=0xf147 waiting on condition

JNI global references: 259

Heap
 PSYoungGen      total 1310208K, used 415069K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)
  eden space 1221632K, 29% used [0x000000076ab00000,0x0000000780d377b8,0x00000007b5400000)
  from space 88576K, 59% used [0x00000007ba980000,0x00000007bdca0030,0x00000007c0000000)
  to   space 87552K, 0% used [0x00000007b5400000,0x00000007b5400000,0x00000007ba980000)
 ParOldGen       total 2796544K, used 575610K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)
  object space 2796544K, 20% used [0x00000006c0000000,0x00000006e321e8f0,0x000000076ab00000)
 Metaspace       used 13998K, capacity 14326K, committed 14592K, reserved 1062912K
  class space    used 1407K, capacity 1473K, committed 1536K, reserved 1048576K




--Chris
________________________________________
From: [hidden email] [[hidden email]] on behalf of Haschart, Robert J. (rh9ec) [[hidden email]]
Sent: Monday, December 19, 2016 6:49 PM
To: [hidden email]; Delis, Christopher; [hidden email]
Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?

One thing that might be useful (if it works) is when the program hangs, do a CRTL-\   (ctrl backslash) or send a QUIT signal  to the process:

kill -3 <pid>

It should cause the program to dump info on all of the threads in the program.

-Bob Haschart
________________________________________
From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 5:00 PM
To: Delis, Christopher; [hidden email]
Cc: [hidden email]
Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?

I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.

I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.

- Demian

-----Original Message-----
From: Delis, Christopher [mailto:[hidden email]]
Sent: Monday, December 19, 2016 4:30 PM
To: [hidden email]
Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel _______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C7de836f1269d40d1958208d428593b02%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=ysxtgzO4X2ocu%2FSBiurQvRrJo%2B8Nr0ypqPPTL%2Fv3MkM%3D&reserved=0

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/solrmarc-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/solrmarc-tech.
For more options, visit https://groups.google.com/d/optout.

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Delis, Christopher
One thing I forgot to mention is that I have two extra lines of commented-out code in IndexDriver.java (lines 151-152), which I forgot to remove.

This will affect the line numbers in the dump below for file IndexDriver.java, e.g., below, instead of line number 295, it should really read line number 293:

"main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
        at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:295)



Here's a new dump with the line numbers fixed:



2016-12-20 15:42:07
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):

"SolrUpdate-6162166-6162823" #23 prio=5 os_prio=0 tid=0x00007fa7885bb000 nid=0xf156 waiting on condition [0x00007fa76c9e3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6162824-6163477" #22 prio=5 os_prio=0 tid=0x00007fa7885ba000 nid=0xf155 waiting on condition [0x00007fa76d426000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6164123-6164771" #21 prio=5 os_prio=0 tid=0x00007fa7885b9000 nid=0xf154 waiting on condition [0x00007fa76d628000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6163478-6164122" #20 prio=5 os_prio=0 tid=0x00007fa7885b3000 nid=0xf153 waiting on condition [0x00007fa76d527000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"RecordIndexer-Thread-0" #19 prio=5 os_prio=0 tid=0x00007fa7885b3800 nid=0xf152 in Object.wait() [0x00007fa76d729000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.solrmarc.driver.IndexerWorker.run(IndexerWorker.java:82)
        - locked <0x00000006c03f5590> (a java.util.concurrent.ArrayBlockingQueue)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"MarcReader-Thread" #18 prio=5 os_prio=0 tid=0x00007fa7885b1800 nid=0xf151 waiting on condition [0x00007fa76d123000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.MarcReaderThread.run(MarcReaderThread.java:40)

"Eclipse-Shutdown-Simulator-Thread" #16 prio=5 os_prio=0 tid=0x00007fa7885ae000 nid=0xf150 waiting on condition [0x00007fa76d224000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.IndexDriver$ShutdownSimulator.run(IndexDriver.java:534)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fa788136800 nid=0xf146 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa788133800 nid=0xf145 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa788130800 nid=0xf144 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa78812e800 nid=0xf143 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa788107000 nid=0xf142 in Object.wait() [0x00007fa76e79c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa788102800 nid=0xf141 in Object.wait() [0x00007fa76e89d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
        at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:293)
        at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:100)
        at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:113)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.Boot.main(Boot.java:72)

"VM Thread" os_prio=0 tid=0x00007fa7880fa800 nid=0xf140 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa78801f800 nid=0xf13e runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa788021000 nid=0xf13f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fa78813d000 nid=0xf147 waiting on condition

JNI global references: 259

Heap
 PSYoungGen      total 1310208K, used 415069K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)
  eden space 1221632K, 29% used [0x000000076ab00000,0x0000000780d377b8,0x00000007b5400000)
  from space 88576K, 59% used [0x00000007ba980000,0x00000007bdca0030,0x00000007c0000000)
  to   space 87552K, 0% used [0x00000007b5400000,0x00000007b5400000,0x00000007ba980000)
 ParOldGen       total 2796544K, used 575610K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)
  object space 2796544K, 20% used [0x00000006c0000000,0x00000006e321e8f0,0x000000076ab00000)
 Metaspace       used 13998K, capacity 14326K, committed 14592K, reserved 1062912K
  class space    used 1407K, capacity 1473K, committed 1536K, reserved 1048576K


________________________________________
From: Delis, Christopher [[hidden email]]
Sent: Tuesday, December 20, 2016 3:45 PM
To: [hidden email]; [hidden email]
Subject: Re: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

In case it helps any, here are the results of the kill -3 <PID>:

2016-12-20 15:42:07
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):

"SolrUpdate-6162166-6162823" #23 prio=5 os_prio=0 tid=0x00007fa7885bb000 nid=0xf156 waiting on condition [0x00007fa76c9e3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6162824-6163477" #22 prio=5 os_prio=0 tid=0x00007fa7885ba000 nid=0xf155 waiting on condition [0x00007fa76d426000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6164123-6164771" #21 prio=5 os_prio=0 tid=0x00007fa7885b9000 nid=0xf154 waiting on condition [0x00007fa76d628000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"SolrUpdate-6163478-6164122" #20 prio=5 os_prio=0 tid=0x00007fa7885b3000 nid=0xf153 waiting on condition [0x00007fa76d527000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"RecordIndexer-Thread-0" #19 prio=5 os_prio=0 tid=0x00007fa7885b3800 nid=0xf152 in Object.wait() [0x00007fa76d729000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.solrmarc.driver.IndexerWorker.run(IndexerWorker.java:82)
        - locked <0x00000006c03f5590> (a java.util.concurrent.ArrayBlockingQueue)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"MarcReader-Thread" #18 prio=5 os_prio=0 tid=0x00007fa7885b1800 nid=0xf151 waiting on condition [0x00007fa76d123000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.MarcReaderThread.run(MarcReaderThread.java:40)

"Eclipse-Shutdown-Simulator-Thread" #16 prio=5 os_prio=0 tid=0x00007fa7885ae000 nid=0xf150 waiting on condition [0x00007fa76d224000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.IndexDriver$ShutdownSimulator.run(IndexDriver.java:536)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fa788136800 nid=0xf146 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa788133800 nid=0xf145 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa788130800 nid=0xf144 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa78812e800 nid=0xf143 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa788107000 nid=0xf142 in Object.wait() [0x00007fa76e79c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa788102800 nid=0xf141 in Object.wait() [0x00007fa76e89d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
        at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:295)
        at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:100)
        at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:113)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
        at org.solrmarc.driver.Boot.main(Boot.java:72)

"VM Thread" os_prio=0 tid=0x00007fa7880fa800 nid=0xf140 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa78801f800 nid=0xf13e runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa788021000 nid=0xf13f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fa78813d000 nid=0xf147 waiting on condition

JNI global references: 259

Heap
 PSYoungGen      total 1310208K, used 415069K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)
  eden space 1221632K, 29% used [0x000000076ab00000,0x0000000780d377b8,0x00000007b5400000)
  from space 88576K, 59% used [0x00000007ba980000,0x00000007bdca0030,0x00000007c0000000)
  to   space 87552K, 0% used [0x00000007b5400000,0x00000007b5400000,0x00000007ba980000)
 ParOldGen       total 2796544K, used 575610K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)
  object space 2796544K, 20% used [0x00000006c0000000,0x00000006e321e8f0,0x000000076ab00000)
 Metaspace       used 13998K, capacity 14326K, committed 14592K, reserved 1062912K
  class space    used 1407K, capacity 1473K, committed 1536K, reserved 1048576K




--Chris
________________________________________
From: [hidden email] [[hidden email]] on behalf of Haschart, Robert J. (rh9ec) [[hidden email]]
Sent: Monday, December 19, 2016 6:49 PM
To: [hidden email]; Delis, Christopher; [hidden email]
Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?

One thing that might be useful (if it works) is when the program hangs, do a CRTL-\   (ctrl backslash) or send a QUIT signal  to the process:

kill -3 <pid>

It should cause the program to dump info on all of the threads in the program.

-Bob Haschart
________________________________________
From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 5:00 PM
To: Delis, Christopher; [hidden email]
Cc: [hidden email]
Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?

I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.

I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.

- Demian

-----Original Message-----
From: Delis, Christopher [mailto:[hidden email]]
Sent: Monday, December 19, 2016 4:30 PM
To: [hidden email]
Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?

[Reposting this from vufind-general because it seems more appropriate here...]

Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.

I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.

Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).

I've upgraded to the latest SolrMarc, but still experience the same hanging issue.

Thanks,
Chris

P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SolrMarc Indexer Hanging? Deadlock issue?

Oliver Obenland
Ok, the pull request is online now:
https://github.com/solrmarc/solrmarc/pull/59

We experienced no hanging with those changes after reindexing everything
4 or 5 times.

- Oliver


Am 20.12.2016 um 23:02 schrieb Delis, Christopher:

> One thing I forgot to mention is that I have two extra lines of commented-out code in IndexDriver.java (lines 151-152), which I forgot to remove.
>
> This will affect the line numbers in the dump below for file IndexDriver.java, e.g., below, instead of line number 295, it should really read line number 293:
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
>          at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:295)
>
>
>
> Here's a new dump with the line numbers fixed:
>
>
>
> 2016-12-20 15:42:07
> Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
>
> "SolrUpdate-6162166-6162823" #23 prio=5 os_prio=0 tid=0x00007fa7885bb000 nid=0xf156 waiting on condition [0x00007fa76c9e3000]
>     java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6162824-6163477" #22 prio=5 os_prio=0 tid=0x00007fa7885ba000 nid=0xf155 waiting on condition [0x00007fa76d426000]
>     java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6164123-6164771" #21 prio=5 os_prio=0 tid=0x00007fa7885b9000 nid=0xf154 waiting on condition [0x00007fa76d628000]
>     java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6163478-6164122" #20 prio=5 os_prio=0 tid=0x00007fa7885b3000 nid=0xf153 waiting on condition [0x00007fa76d527000]
>     java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "RecordIndexer-Thread-0" #19 prio=5 os_prio=0 tid=0x00007fa7885b3800 nid=0xf152 in Object.wait() [0x00007fa76d729000]
>     java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.solrmarc.driver.IndexerWorker.run(IndexerWorker.java:82)
> - locked <0x00000006c03f5590> (a java.util.concurrent.ArrayBlockingQueue)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> "MarcReader-Thread" #18 prio=5 os_prio=0 tid=0x00007fa7885b1800 nid=0xf151 waiting on condition [0x00007fa76d123000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.solrmarc.driver.MarcReaderThread.run(MarcReaderThread.java:40)
>
> "Eclipse-Shutdown-Simulator-Thread" #16 prio=5 os_prio=0 tid=0x00007fa7885ae000 nid=0xf150 waiting on condition [0x00007fa76d224000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.solrmarc.driver.IndexDriver$ShutdownSimulator.run(IndexDriver.java:534)
>
> "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fa788136800 nid=0xf146 runnable [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa788133800 nid=0xf145 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa788130800 nid=0xf144 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa78812e800 nid=0xf143 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa788107000 nid=0xf142 in Object.wait() [0x00007fa76e79c000]
>     java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa788102800 nid=0xf141 in Object.wait() [0x00007fa76e89d000]
>     java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
> at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:293)
> at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:100)
> at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
> at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
> at org.solrmarc.driver.Boot.main(Boot.java:72)
>
> "VM Thread" os_prio=0 tid=0x00007fa7880fa800 nid=0xf140 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa78801f800 nid=0xf13e runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa788021000 nid=0xf13f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fa78813d000 nid=0xf147 waiting on condition
>
> JNI global references: 259
>
> Heap
>   PSYoungGen      total 1310208K, used 415069K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)
>    eden space 1221632K, 29% used [0x000000076ab00000,0x0000000780d377b8,0x00000007b5400000)
>    from space 88576K, 59% used [0x00000007ba980000,0x00000007bdca0030,0x00000007c0000000)
>    to   space 87552K, 0% used [0x00000007b5400000,0x00000007b5400000,0x00000007ba980000)
>   ParOldGen       total 2796544K, used 575610K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)
>    object space 2796544K, 20% used [0x00000006c0000000,0x00000006e321e8f0,0x000000076ab00000)
>   Metaspace       used 13998K, capacity 14326K, committed 14592K, reserved 1062912K
>    class space    used 1407K, capacity 1473K, committed 1536K, reserved 1048576K
>
>
> ________________________________________
> From: Delis, Christopher [[hidden email]]
> Sent: Tuesday, December 20, 2016 3:45 PM
> To: [hidden email]; [hidden email]
> Subject: Re: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?
>
> In case it helps any, here are the results of the kill -3 <PID>:
>
> 2016-12-20 15:42:07
> Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
>
> "SolrUpdate-6162166-6162823" #23 prio=5 os_prio=0 tid=0x00007fa7885bb000 nid=0xf156 waiting on condition [0x00007fa76c9e3000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6162824-6163477" #22 prio=5 os_prio=0 tid=0x00007fa7885ba000 nid=0xf155 waiting on condition [0x00007fa76d426000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6164123-6164771" #21 prio=5 os_prio=0 tid=0x00007fa7885b9000 nid=0xf154 waiting on condition [0x00007fa76d628000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> "SolrUpdate-6163478-6164122" #20 prio=5 os_prio=0 tid=0x00007fa7885b3000 nid=0xf153 waiting on condition [0x00007fa76d527000]
>     java.lang.Thread.State: WAITING (parking)
>          at sun.misc.Unsafe.park(Native Method)
>          - parking to wait for  <0x00000006c19daf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> "RecordIndexer-Thread-0" #19 prio=5 os_prio=0 tid=0x00007fa7885b3800 nid=0xf152 in Object.wait() [0x00007fa76d729000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          at java.lang.Object.wait(Object.java:502)
>          at org.solrmarc.driver.IndexerWorker.run(IndexerWorker.java:82)
>          - locked <0x00000006c03f5590> (a java.util.concurrent.ArrayBlockingQueue)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> "MarcReader-Thread" #18 prio=5 os_prio=0 tid=0x00007fa7885b1800 nid=0xf151 waiting on condition [0x00007fa76d123000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at org.solrmarc.driver.MarcReaderThread.run(MarcReaderThread.java:40)
>
> "Eclipse-Shutdown-Simulator-Thread" #16 prio=5 os_prio=0 tid=0x00007fa7885ae000 nid=0xf150 waiting on condition [0x00007fa76d224000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at org.solrmarc.driver.IndexDriver$ShutdownSimulator.run(IndexDriver.java:536)
>
> "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fa788136800 nid=0xf146 runnable [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fa788133800 nid=0xf145 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fa788130800 nid=0xf144 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fa78812e800 nid=0xf143 waiting on condition [0x0000000000000000]
>     java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fa788107000 nid=0xf142 in Object.wait() [0x00007fa76e79c000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>          - locked <0x00000006c03a72d0> (a java.lang.ref.ReferenceQueue$Lock)
>          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fa788102800 nid=0xf141 in Object.wait() [0x00007fa76e89d000]
>     java.lang.Thread.State: WAITING (on object monitor)
>          at java.lang.Object.wait(Native Method)
>          - waiting on <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
>          at java.lang.Object.wait(Object.java:502)
>          at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
>          - locked <0x00000006c14f2850> (a java.lang.ref.Reference$Lock)
>          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007fa78800a000 nid=0xf13d waiting on condition [0x00007fa78fe9f000]
>     java.lang.Thread.State: TIMED_WAITING (sleeping)
>          at java.lang.Thread.sleep(Native Method)
>          at org.solrmarc.driver.ThreadedIndexer.indexToSolr(ThreadedIndexer.java:196)
>          at org.solrmarc.driver.IndexDriver.processInput(IndexDriver.java:295)
>          at org.solrmarc.driver.IndexDriver.execute(IndexDriver.java:100)
>          at org.solrmarc.driver.IndexDriver.main(IndexDriver.java:66)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
>          at org.solrmarc.driver.ConfigDriver.main(ConfigDriver.java:113)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.solrmarc.driver.Boot.invokeMain(Boot.java:91)
>          at org.solrmarc.driver.Boot.main(Boot.java:72)
>
> "VM Thread" os_prio=0 tid=0x00007fa7880fa800 nid=0xf140 runnable
>
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fa78801f800 nid=0xf13e runnable
>
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fa788021000 nid=0xf13f runnable
>
> "VM Periodic Task Thread" os_prio=0 tid=0x00007fa78813d000 nid=0xf147 waiting on condition
>
> JNI global references: 259
>
> Heap
>   PSYoungGen      total 1310208K, used 415069K [0x000000076ab00000, 0x00000007c0000000, 0x00000007c0000000)
>    eden space 1221632K, 29% used [0x000000076ab00000,0x0000000780d377b8,0x00000007b5400000)
>    from space 88576K, 59% used [0x00000007ba980000,0x00000007bdca0030,0x00000007c0000000)
>    to   space 87552K, 0% used [0x00000007b5400000,0x00000007b5400000,0x00000007ba980000)
>   ParOldGen       total 2796544K, used 575610K [0x00000006c0000000, 0x000000076ab00000, 0x000000076ab00000)
>    object space 2796544K, 20% used [0x00000006c0000000,0x00000006e321e8f0,0x000000076ab00000)
>   Metaspace       used 13998K, capacity 14326K, committed 14592K, reserved 1062912K
>    class space    used 1407K, capacity 1473K, committed 1536K, reserved 1048576K
>
>
>
>
> --Chris
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Haschart, Robert J. (rh9ec) [[hidden email]]
> Sent: Monday, December 19, 2016 6:49 PM
> To: [hidden email]; Delis, Christopher; [hidden email]
> Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?
>
> One thing that might be useful (if it works) is when the program hangs, do a CRTL-\   (ctrl backslash) or send a QUIT signal  to the process:
>
> kill -3 <pid>
>
> It should cause the program to dump info on all of the threads in the program.
>
> -Bob Haschart
> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
> Sent: Monday, December 19, 2016 5:00 PM
> To: Delis, Christopher; [hidden email]
> Cc: [hidden email]
> Subject: [solrmarc-tech] RE: SolrMarc Indexer Hanging? Deadlock issue?
>
> I haven't encountered this problem myself, but I don't have nearly as many records as you. I'm also cross-posting this to solrmarc-tech to reach a broader audience.
>
> I do know that some of the earlier 3.x releases had trouble handling errors and would seemingly hang when they were supposed to output error messages... that should be fixed in newer releases.
>
> - Demian
>
> -----Original Message-----
> From: Delis, Christopher [mailto:[hidden email]]
> Sent: Monday, December 19, 2016 4:30 PM
> To: [hidden email]
> Subject: [VuFind-Tech] SolrMarc Indexer Hanging? Deadlock issue?
>
> [Reposting this from vufind-general because it seems more appropriate here...]
>
> Has anyone else noticed SolrMarc hang during indexing? It appears to happen randomly (i.e., it doesn't fail on a particular record or anything like that). I'm suspecting that it might be a deadlock issue in the multi-threaded processing, but I haven't dug deeper into it yet. I was just wondering if I was the only one experiencing this.
>
> I do index very large sets of records (tens of millions) and the hanging tends to occur every couple of million records or so (e.g., during a recent indexing of 35 million records, it occurred 10 times). The MARC record files are at most 1GB each.
>
> Today, I am attempting to index authority records for the first time and I'm having even worse luck; I can't seem to load a single MARC record file (1GB) without it hanging (sometimes after ~700,000 records, sometimes after a ~million).
>
> I've upgraded to the latest SolrMarc, but still experience the same hanging issue.
>
> Thanks,
> Chris
>
> P.S., I'm currently running SolrMarc in single-threaded mode, and so far it seems to run without hanging (I'm still indexing records at the moment). But it's noticeably slower (obviously due to the single threaded-ness).
>
>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
> _______________________________________________
> Vufind-tech mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/vufind-tech


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Loading...