indexing with -p flag after upgrade

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

indexing with -p flag after upgrade

Bradley Busenius
We're in the process of upgrading to VuFind 3.1. Things have gone well,
however, we're not able to index using the -p flag since we've upgraded.
The following command fails.

./import-marc.sh -p local/import/import_ole.properties
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom,
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: indexing with -p flag after upgrade

Demian Katz

Brad,


Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:


1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).


2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.


The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)


- Demian



From: Brad Busenius <[hidden email]>
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade
 
We're in the process of upgrading to VuFind 3.1. Things have gone well,
however, we're not able to index using the -p flag since we've upgraded.
The following command fails.

./import-marc.sh -p local/import/import_ole.properties
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom,
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: indexing with -p flag after upgrade

Bradley Busenius

Hi Demian,


Yes, we have a rule like this in our marc.properties:

title_full = custom, getAllAlphaExcept(245c)

When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:

topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")


series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)


title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)
When I comment all of these out, our import is able to complete.

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help!

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:

Brad,


Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:


1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).


2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.


The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)


- Demian



From: Brad Busenius [hidden email]
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade
 
We're in the process of upgrading to VuFind 3.1. Things have gone well,
however, we're not able to index using the -p flag since we've upgraded.
The following command fails.

./import-marc.sh -p local/import/import_ole.properties
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom,
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: indexing with -p flag after upgrade

Demian Katz

It looks like the new index specification language supports ranges, so I wonder if:

 

title_full = 245[ac-z0-9]

 

would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).

 

It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.

 

I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?

 

Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.

 

- Demian

 

From: Brad Busenius [mailto:[hidden email]]
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Hi Demian,

 

Yes, we have a rule like this in our marc.properties:

title_full = custom, getAllAlphaExcept(245c)

When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:

topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")

 

series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)


title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete.

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help!

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:

Brad,

 

Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:

 

1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).

 

2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.

 

The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)

 

- Demian

 


From: Brad Busenius [hidden email]
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade

 

We're in the process of upgrading to VuFind 3.1. Things have gone well,
however, we're not able to index using the -p flag since we've upgraded.
The following command fails.

./import-marc.sh -p local/import/import_ole.properties
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom,
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: indexing with -p flag after upgrade

Demian Katz

Brad,

 

The custom_map error has something to do with the index specification parser. Are you running the latest version of SolrMarc? I think some bugs were fixed in here, so if you’re not already on the newest version, upgrading the .jar files might make a difference.

 

Regarding your custom Beanshell, there’s a lot going on in there -- it might be useful to find out some history about this and see if it’s all still needed. I’d guess that some of this should be able to be simplified now. At a glance, one problem that might be causing issues is the use of static references to the SolrIndexer class. See this diff to one of the existing VuFind BeanShell scripts for a suggestion on how to fix it:

 

https://github.com/vufind-org/vufind/commit/72766a05844c3bb4f5ca58b236d3c762320c0843

 

(Basically you should be able to simply replace SolrIndexer. with index.)

 

- Demian

 

From: Brad Busenius [mailto:[hidden email]]
Sent: Tuesday, January 03, 2017 12:42 PM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Your title_full example seems to have worked. I haven't looked at the data yet but it didn't throw any errors. The error for getAllAlphaExcept is as follows:

 

Malformed custom_map reference : Should be
custom_map(fully.qualified.class.Name method)
 or  custom_map(fully.qualified.class.Name method(parm1, parm2))


The ucTitleBrowseFunctions.bsh script is attached. I'm looking it over now too.

-Brad

On 1/3/17 11:13 AM, Demian Katz wrote:

It looks like the new index specification language supports ranges, so I wonder if:

 

title_full = 245[ac-z0-9]

 

would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).

 

It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.

 

I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?

 

Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.

 

- Demian

 

From: Brad Busenius [[hidden email]]
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Hi Demian,

 

Yes, we have a rule like this in our marc.properties:

title_full = custom, getAllAlphaExcept(245c)

When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:

topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")

 

series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)


title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete.

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help!

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:

Brad,

 

Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:

 

1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).

 

2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.

 

The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)

 

- Demian

 


From: Brad Busenius [hidden email]
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade

 

We're in the process of upgrading to VuFind 3.1. Things have gone well,
however, we're not able to index using the -p flag since we've upgraded.
The following command fails.

./import-marc.sh -p local/import/import_ole.properties
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom,
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

 

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Trim non-filling data

Sean Filipov


Vufind-techList,

We’re moving from VuFind 2.4 to VuFind 3.1 with Solr 5.5.0.

Our title browse pulls from many fields and includes versions of the title with and without the non-filing prefix. The old way, we checked for non-filling indicator, trimmed the data and had one version of trimmed data and one version of non-trimmed data was by modifying the record in Beanshell. If we got, for an example, The Hobbit, the output would have to versions The Hobbit , and Hobbit (without article) .

How to do that with the new SolrMarc?



 * 
 * Example:
 * Input: 
 *     245 _4 |a The Hobbit
 * Output: 
 *     245 _4 |a The Hobbit
 *     245 _0 |a The Hobbit






Brad,
 
The custom_map error has something to do with the index specification parser. Are you running the latest version of SolrMarc? I think some bugs were fixed in here, so if you’re not already on the newest version, upgrading the .jar files might make a difference.
 
Regarding your custom Beanshell, there’s a lot going on in there -- it might be useful to find out some history about this and see if it’s all still needed. I’d guess that some of this should be able to be simplified now. At a glance, one problem that might be causing issues is the use of static references to the SolrIndexer class. See this diff to one of the existing VuFind BeanShell scripts for a suggestion on how to fix it:
 
 
(Basically you should be able to simply replace SolrIndexer. with index.)
 
- Demian
 
From: Brad Busenius [[hidden email]] 
Sent: Tuesday, January 03, 2017 12:42 PM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade
 
Your title_full example seems to have worked. I haven't looked at the data yet but it didn't throw any errors. The error for getAllAlphaExcept is as follows:
 
Malformed custom_map reference : Should be
custom_map(fully.qualified.class.Name method)
 or  custom_map(fully.qualified.class.Name method(parm1, parm2))


The ucTitleBrowseFunctions.bsh script is attached. I'm looking it over now too. 

-Brad

On 1/3/17 11:13 AM, Demian Katz wrote:
It looks like the new index specification language supports ranges, so I wonder if:
 
title_full = 245[ac-z0-9]
 
would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).
 
It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.
 
I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?
 
Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.
 
- Demian
 
From: Brad Busenius [[hidden email]] 
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade
 
Hi Demian,
 
Yes, we have a rule like this in our marc.properties: 
title_full = custom, getAllAlphaExcept(245c)
When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:
topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")
 
series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)

title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete. 

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help! 

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:
Brad,
 
Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:
 
1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).
 
2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.
 
The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)
 
- Demian
 

From: Brad Busenius [hidden email]
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade
 
We're in the process of upgrading to VuFind 3.1. Things have gone well, 
however, we're not able to index using the -p flag since we've upgraded. 
The following command fails.

./import-marc.sh -p local/import/import_ole.properties 
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom, 
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor 
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class 
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0
 
 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Trim non-filling data

Demian Katz

Sean,

 

The new SolrMarc introduces a += operator, so you can do things like:

 

title_browse = 245a

title_browse += custom, myCustomMethod

title_browse += 999z

 

I suspect that this may allow you to achieve the desired result in a much simpler fashion.

 

Also note that the new SolrMarc still supports BeanShell, so it should also be possible to make your old code continue to work, though it’s possible minor syntactic adjustments may be necessary.

 

I hope this helps – but please let us know if questions remain!

 

- Demian

 

From: Sean Filipov [mailto:[hidden email]]
Sent: Wednesday, January 04, 2017 4:25 PM
To: [hidden email]; [hidden email]
Subject: [VuFind-Tech] Trim non-filling data

 

 

 

Vufind-techList,

 

We’re moving from VuFind 2.4 to VuFind 3.1 with Solr 5.5.0.

 

Our title browse pulls from many fields and includes versions of the title with and without the non-filing prefix. The old way, we checked for non-filling indicator, trimmed the data and had one version of trimmed data and one version of non-trimmed data was by modifying the record in Beanshell. If we got, for an example, The Hobbit, the output would have to versions The Hobbit , and Hobbit (without article) .

 

How to do that with the new SolrMarc?

 

 

 

 * 

 * Example:

 * Input: 

 *     245 _4 |a The Hobbit

 * Output: 

 *     245 _4 |a The Hobbit

 *     245 _0 |a The Hobbit

 

 

 

 

 

 

Brad,

 

The custom_map error has something to do with the index specification parser. Are you running the latest version of SolrMarc? I think some bugs were fixed in here, so if you’re not already on the newest version, upgrading the .jar files might make a difference.

 

Regarding your custom Beanshell, there’s a lot going on in there -- it might be useful to find out some history about this and see if it’s all still needed. I’d guess that some of this should be able to be simplified now. At a glance, one problem that might be causing issues is the use of static references to the SolrIndexer class. See this diff to one of the existing VuFind BeanShell scripts for a suggestion on how to fix it:

 

 

(Basically you should be able to simply replace SolrIndexer. with index.)

 

- Demian

 

From: Brad Busenius [[hidden email]] 
Sent: Tuesday, January 03, 2017 12:42 PM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Your title_full example seems to have worked. I haven't looked at the data yet but it didn't throw any errors. The error for getAllAlphaExcept is as follows:

 

Malformed custom_map reference : Should be
custom_map(fully.qualified.class.Name method)
 or  custom_map(fully.qualified.class.Name method(parm1, parm2))


The ucTitleBrowseFunctions.bsh script is attached. I'm looking it over now too. 

-Brad


On 1/3/17 11:13 AM, Demian Katz wrote:

It looks like the new index specification language supports ranges, so I wonder if:

 

title_full = 245[ac-z0-9]

 

would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).

 

It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.

 

I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?

 

Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.

 

- Demian

 

From: Brad Busenius [[hidden email]] 
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; [hidden email]
Cc: [hidden email]
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Hi Demian,

 

Yes, we have a rule like this in our marc.properties: 

title_full = custom, getAllAlphaExcept(245c)

When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:

topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")

 

series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)


title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete. 

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help! 

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:

Brad,

 

Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:

 

1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).

 

2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.

 

The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)

 

- Demian

 


From: Brad Busenius [hidden email]
Sent: Wednesday, December 28, 2016 2:10 PM
To: [hidden email]
Subject: [VuFind-Tech] indexing with -p flag after upgrade

 

We're in the process of upgrading to VuFind 3.1. Things have gone well, 
however, we're not able to index using the -p flag since we've upgraded. 
The following command fails.

./import-marc.sh -p local/import/import_ole.properties 
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom, 
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor 
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class 
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

 

 

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Trim non-filling data

Robert Haschart
Sean,

I have good news, and bad news.   Doing what you want is fairly easy, using the   +=  syntax Demian mentions,  you can simply do the following:

title_browse = 245ab, clean
title_browse += 245ab?(ind2 > 0), stripInd2, clean

This will add the title field, and then only add then title field again with the non-filing characters removed, only if the indicator2 value is greater than zero.

However the bad news is that the greater than operator (>)  and the less than operator (<)  were both implemented backwards.   D'oh. 

So in the currently released version you could do this:

title_browse = 245ab, clean
title_browse += 245ab?(ind2 < 0), stripInd2, clean

and it would work, but it would break when the new release comes out fixing that bug.  Or I guess this should work as well.

title_browse = 245ab, clean
title_browse += 245ab?(ind2 != 0), stripInd2, clean

I just tested it, and for a record with title field:

245 04$aThe Massachusetts spy$h[microform]

the above lines with !=  produces:

title_browse : The Massachusetts spy
title_browse : Massachusetts spy

but with the title field:

245 00$aSylvia Marlowe, harpsichord.$h[sound recording].

it produces:

title_browse : Sylvia Marlowe, harpsichord

-Bob Haschart





From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
Sent: Thursday, January 05, 2017 9:34 AM
To: Sean Filipov; [hidden email]; [hidden email]
Subject: [solrmarc-tech] RE: Trim non-filling data

Sean,

 

The new SolrMarc introduces a += operator, so you can do things like:

 

title_browse = 245a

title_browse += custom, myCustomMethod

title_browse += 999z

 

I suspect that this may allow you to achieve the desired result in a much simpler fashion.

 

Also note that the new SolrMarc still supports BeanShell, so it should also be possible to make your old code continue to work, though it’s possible minor syntactic adjustments may be necessary.

 

I hope this helps – but please let us know if questions remain!

 

- Demian

 

From: Sean Filipov [mailto:[hidden email]]
Sent: Wednesday, January 04, 2017 4:25 PM
To: [hidden email]; [hidden email]
Subject: [VuFind-Tech] Trim non-filling data

 

 

 

Vufind-techList,

 

We’re moving from VuFind 2.4 to VuFind 3.1 with Solr 5.5.0.

 

Our title browse pulls from many fields and includes versions of the title with and without the non-filing prefix. The old way, we checked for non-filling indicator, trimmed the data and had one version of trimmed data and one version of non-trimmed data was by modifying the record in Beanshell. If we got, for an example, The Hobbit, the output would have to versions The Hobbit , and Hobbit (without article) .

 

How to do that with the new SolrMarc?

 

 

 

 * 

 * Example:

 * Input: 

 *     245 _4 |a The Hobbit

 * Output: 

 *     245 _4 |a The Hobbit

 *     245 _0 |a The Hobbit

 

 

 

 

 

 

Brad,

 

The custom_map error has something to do with the index specification parser. Are you running the latest version of SolrMarc? I think some bugs were fixed in here, so if you’re not already on the newest version, upgrading the .jar files might make a difference.

 

Regarding your custom Beanshell, there’s a lot going on in there -- it might be useful to find out some history about this and see if it’s all still needed. I’d guess that some of this should be able to be simplified now. At a glance, one problem that might be causing issues is the use of static references to the SolrIndexer class. See this diff to one of the existing VuFind BeanShell scripts for a suggestion on how to fix it:

 

 

(Basically you should be able to simply replace SolrIndexer. with index.)

 

- Demian

 

From: Brad Busenius [mailto:bbusenius@...] 
Sent: Tuesday, January 03, 2017 12:42 PM
To: Demian Katz; vufind-tech@...
Cc: solrmarc-tech@...
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Your title_full example seems to have worked. I haven't looked at the data yet but it didn't throw any errors. The error for getAllAlphaExcept is as follows:

 

Malformed custom_map reference : Should be
custom_map(fully.qualified.class.Name method)
 or  custom_map(fully.qualified.class.Name method(parm1, parm2))


The ucTitleBrowseFunctions.bsh script is attached. I'm looking it over now too. 

-Brad


On 1/3/17 11:13 AM, Demian Katz wrote:

It looks like the new index specification language supports ranges, so I wonder if:

 

title_full = 245[ac-z0-9]

 

would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).

 

It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.

 

I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?

 

Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.

 

- Demian

 

From: Brad Busenius [mailto:bbusenius@...] 
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; vufind-tech@...
Cc: solrmarc-tech@...
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Hi Demian,

 

Yes, we have a rule like this in our marc.properties: 

title_full = custom, getAllAlphaExcept(245c)

When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:

topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")

 

series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)


title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete. 

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help! 

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:

Brad,

 

Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:

 

1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).

 

2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.

 

The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)

 

- Demian

 


From: Brad Busenius <bbusenius@...>
Sent: Wednesday, December 28, 2016 2:10 PM
To: vufind-tech@...
Subject: [VuFind-Tech] indexing with -p flag after upgrade

 

We're in the process of upgrading to VuFind 3.1. Things have gone well, 
however, we're not able to index using the -p flag since we've upgraded. 
The following command fails.

./import-marc.sh -p local/import/import_ole.properties 
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom, 
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor 
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class 
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
Vufind-tech@...
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

 

 

 

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tech+unsubscribe@....
To post to this group, send email to solrmarc-tech@....
Visit this group at https://groups.google.com/group/solrmarc-tech.
For more options, visit https://groups.google.com/d/optout.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Trim non-filling data

Sean Filipov
Yes, it worked.
Thank you




On Jan 5, 2017, at 11:44 AM, Haschart, Robert J. (rh9ec) <[hidden email]> wrote:

Sean,

I have good news, and bad news.   Doing what you want is fairly easy, using the   +=  syntax Demian mentions,  you can simply do the following:

title_browse = 245ab, clean
title_browse += 245ab?(ind2 > 0), stripInd2, clean

This will add the title field, and then only add then title field again with the non-filing characters removed, only if the indicator2 value is greater than zero.

However the bad news is that the greater than operator (>)  and the less than operator (<)  were both implemented backwards.   D'oh.  

So in the currently released version you could do this:

title_browse = 245ab, clean
title_browse += 245ab?(ind2 < 0), stripInd2, clean

and it would work, but it would break when the new release comes out fixing that bug.  Or I guess this should work as well.

title_browse = 245ab, clean
title_browse += 245ab?(ind2 != 0), stripInd2, clean

I just tested it, and for a record with title field:

245 04$aThe Massachusetts spy$h[microform]

the above lines with !=  produces:

title_browse : The Massachusetts spy
title_browse : Massachusetts spy

but with the title field:

245 00$aSylvia Marlowe, harpsichord.$h[sound recording].

it produces:

title_browse : Sylvia Marlowe, harpsichord

-Bob Haschart





From: [hidden email] [[hidden email]] on behalf of Demian Katz [[hidden email]]
Sent: Thursday, January 05, 2017 9:34 AM
To: Sean Filipov; [hidden email]; [hidden email]
Subject: [solrmarc-tech] RE: Trim non-filling data

Sean,

 

The new SolrMarc introduces a += operator, so you can do things like:

 

title_browse = 245a
title_browse += custom, myCustomMethod
title_browse += 999z

 

I suspect that this may allow you to achieve the desired result in a much simpler fashion.

 

Also note that the new SolrMarc still supports BeanShell, so it should also be possible to make your old code continue to work, though it’s possible minor syntactic adjustments may be necessary.

 

I hope this helps – but please let us know if questions remain!

 

- Demian

 

From: Sean Filipov [[hidden email]] 
Sent: Wednesday, January 04, 2017 4:25 PM
To: [hidden email]; [hidden email]
Subject: [VuFind-Tech] Trim non-filling data

 

 

 

Vufind-techList,

 

We’re moving from VuFind 2.4 to VuFind 3.1 with Solr 5.5.0.

 

Our title browse pulls from many fields and includes versions of the title with and without the non-filing prefix. The old way, we checked for non-filling indicator, trimmed the data and had one version of trimmed data and one version of non-trimmed data was by modifying the record in Beanshell. If we got, for an example, The Hobbit, the output would have to versions The Hobbit , and Hobbit (without article) .

 

How to do that with the new SolrMarc?

 

 

 

 * 
 * Example:
 * Input: 
 *     245 _4 |a The Hobbit
 * Output: 
 *     245 _4 |a The Hobbit
 *     245 _0 |a The Hobbit

 

 

 

 

 

 

Brad,

 

The custom_map error has something to do with the index specification parser. Are you running the latest version of SolrMarc? I think some bugs were fixed in here, so if you’re not already on the newest version, upgrading the .jar files might make a difference.

 

Regarding your custom Beanshell, there’s a lot going on in there -- it might be useful to find out some history about this and see if it’s all still needed. I’d guess that some of this should be able to be simplified now. At a glance, one problem that might be causing issues is the use of static references to the SolrIndexer class. See this diff to one of the existing VuFind BeanShell scripts for a suggestion on how to fix it:

 

<a href="x-msg://2/redir.aspx?REF=OYe_Z6JBxCGCO2hL3Cic0VeEhLANt0zLZ79kxwOt5uzllhuQkDXUCAFodHRwczovL25hMDEuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwcyUzQSUyRiUyRmdpdGh1Yi5jb20lMkZ2dWZpbmQtb3JnJTJGdnVmaW5kJTJGY29tbWl0JTJGNzI3NjZhMDU4NDRjM2JiNGY1Y2E1OGIyMzZkM2M3NjIzMjBjMDg0MyZkYXRhPTAxJTdDMDElN0NkZW1pYW4ua2F0eiU0MHZpbGxhbm92YS5lZHUlN0M0OWE1MGY0NTllMGU0ZjNlNjQ5NzA4ZDQzNGU4NmE4MyU3Qzc2NWE4ZGU1Y2Y5NDQ0ZjA5Y2FmYWU1YmY4Y2ZhMzY2JTdDMSZzZGF0YT1RbTE5S3RXb210OTcxbDdmZUJGbUg3STJ4b05TSktZWmQ3ak9vTGQ0WVFvJTNEJnJlc2VydmVkPTA." target="_blank" style="color: purple; text-decoration: underline;" class="">https://github.com/vufind-org/vufind/commit/72766a05844c3bb4f5ca58b236d3c762320c0843

 

(Basically you should be able to simply replace SolrIndexer. with index.)

 

- Demian

 

From: Brad Busenius [<a href="x-msg://2/redir.aspx?REF=5L6peIlUH33blZEMZ-8ugNCjEKJAARZ2njZxEmZ6hzbllhuQkDXUCAFtYWlsdG86YmJ1c2VuaXVzQHVjaGljYWdvLmVkdQ.." target="_blank" style="color: purple; text-decoration: underline;" class="">mailto:bbusenius@...] 
Sent: Tuesday, January 03, 2017 12:42 PM
To: Demian Katz; <a href="x-msg://2/redir.aspx?REF=vwEJRbZSoyQbw1rjOlTJctyXdY2vJyCs4-xteOLWa27llhuQkDXUCAFtYWlsdG86dnVmaW5kLXRlY2hAbGlzdHMuc291cmNlZm9yZ2UubmV0" target="_blank" style="color: purple; text-decoration: underline;" class="">vufind-tech@...
Cc: <a href="x-msg://2/redir.aspx?REF=9LjZ-MeaQf5LCnL_ECPHxM3-zk9VOQNNKHVF9_6u0xnllhuQkDXUCAFtYWlsdG86c29scm1hcmMtdGVjaEBnb29nbGVncm91cHMuY29t" target="_blank" style="color: purple; text-decoration: underline;" class="">solrmarc-tech@...
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Your title_full example seems to have worked. I haven't looked at the data yet but it didn't throw any errors. The error for getAllAlphaExcept is as follows:

 

Malformed custom_map reference : Should be
custom_map(fully.qualified.class.Name method)
 or  custom_map(fully.qualified.class.Name method(parm1, parm2))


The ucTitleBrowseFunctions.bsh script is attached. I'm looking it over now too. 

-Brad


On 1/3/17 11:13 AM, Demian Katz wrote:
It looks like the new index specification language supports ranges, so I wonder if:

 

title_full = 245[ac-z0-9]

 

would serve to replace the getAllAlphaExcept call. (I haven’t tested this, so I’m just assuming that this is using regular expression range syntax – if I’m mistaken about this, the example may not work).

 

It does seem like it might be useful to reimplement getAllAlphaExcept for backward compatibility purposes, especially if it’s trivially easy to do so. Bob can probably comment on that better than I can.

 

I’m not sure why getAllAlphaSubfields is failing – that one does appear to be implemented in the new SolrMarc code. What error are you seeing in association with that?

 

Regarding your browse errors, could you share your custom .bsh script? Perhaps there is a minor syntactic adjustment needed there for compatibility with the new code.

 

- Demian

 

From: Brad Busenius [<a href="x-msg://2/redir.aspx?REF=5L6peIlUH33blZEMZ-8ugNCjEKJAARZ2njZxEmZ6hzbllhuQkDXUCAFtYWlsdG86YmJ1c2VuaXVzQHVjaGljYWdvLmVkdQ.." target="_blank" style="color: purple; text-decoration: underline;" class="">mailto:bbusenius@...] 
Sent: Tuesday, January 03, 2017 11:34 AM
To: Demian Katz; <a href="x-msg://2/redir.aspx?REF=vwEJRbZSoyQbw1rjOlTJctyXdY2vJyCs4-xteOLWa27llhuQkDXUCAFtYWlsdG86dnVmaW5kLXRlY2hAbGlzdHMuc291cmNlZm9yZ2UubmV0" target="_blank" style="color: purple; text-decoration: underline;" class="">vufind-tech@...
Cc: <a href="x-msg://2/redir.aspx?REF=9LjZ-MeaQf5LCnL_ECPHxM3-zk9VOQNNKHVF9_6u0xnllhuQkDXUCAFtYWlsdG86c29scm1hcmMtdGVjaEBnb29nbGVncm91cHMuY29t" target="_blank" style="color: purple; text-decoration: underline;" class="">solrmarc-tech@...
Subject: Re: [VuFind-Tech] indexing with -p flag after upgrade

 

Hi Demian,

 

Yes, we have a rule like this in our marc.properties: 
title_full = custom, getAllAlphaExcept(245c)
When I change it to: title_full = 245ab we get farther in the indexing process. Apparently we have problems with these others as well:
topic = custom, getAllAlphaSubfields(600:610:611:630:648:650:651:653:654:655:656:662, " ")

 

series_browse = script(ucTitleBrowseFunctions.bsh), getSortableFieldsAsSet(400abcdfgklnptu:410abcdfgklnptu:411acdefgklnpqtu:440anp:490a:LNK400abcdfgklnptu:LNK410abcdfgklnptu:LNK411acdefgklnpqtu:LNK440anp:LNK490a:800abcdfghjklmnopqrstu:810abcdfghklmnoprstu:811acdefghklnpqstu:830adfghklmnoprst:840ah:LNK800abcdfghjklmnopqrstu:LNK810abcdfghklmnoprstu:LNK811acdefghklnpqstu:LNK830adfghklmnoprst:LNK840ah)

title_browse_sort = script(ucTitleBrowseFunctions.bsh), titleBrowseSort(130adfklmnoprs:210ab:240adfklmnoprs:243adfklmnoprs:245abfgknps:246abnp:247abnp:534t:700fklmnoprst:710fklmnoprst:711fklnpst:730adfklmnoprs:740anp:775t:776t:LNK130adfklmnoprs:LNK210ab:LNK240adfklmnoprs:LNK243adfklmnoprs:LNK245abfgknps:LNK246abnp:LNK247abnp:LNK534t:LNK700fklmnoprst:LNK710fklmnoprst:LNK711fklnpst:LNK730adfklmnoprs:LNK740anp:LNK775t:LNK776t)

When I comment all of these out, our import is able to complete. 

We're going to continue looking at topic, series_browse, and title_browse_sort. I imagine we just need to change our syntax. It's the getAllAlphaExcept that we don't know how to handle. When I looked at the documentation, I didn't see an immediate replacement. Anyhow, please forgive my ignorance. I haven't dealt with any of this indexing stuff before.

As always, thanks for your help! 

- Brad

On 12/29/16 7:01 AM, Demian Katz wrote:
Brad,

 

Do you have a custom import rule using getAllAlphaExcept? At a glance, there are two possibilities:

 

1.) We didn't reimplement -p correctly when we upgraded to SolrMarc 3 (possible, because I don't believe I've tested that scenario).

 

2.) SolrMarc 3 doesn't implement getAllAlphaExcept properly.

 

The answer to my question may narrow down which it is. Let me know, and I'll see if I can help come up with a workaround. I'm also copying solrmarc-tech in case Bob gets there first. :-)

 

- Demian

 


From: Brad Busenius <a href="x-msg://2/redir.aspx?REF=5L6peIlUH33blZEMZ-8ugNCjEKJAARZ2njZxEmZ6hzbllhuQkDXUCAFtYWlsdG86YmJ1c2VuaXVzQHVjaGljYWdvLmVkdQ.." target="_blank" style="color: purple; text-decoration: underline;" class=""><bbusenius@...>
Sent: Wednesday, December 28, 2016 2:10 PM
To: <a href="x-msg://2/redir.aspx?REF=vwEJRbZSoyQbw1rjOlTJctyXdY2vJyCs4-xteOLWa27llhuQkDXUCAFtYWlsdG86dnVmaW5kLXRlY2hAbGlzdHMuc291cmNlZm9yZ2UubmV0" target="_blank" style="color: purple; text-decoration: underline;" class="">vufind-tech@...
Subject: [VuFind-Tech] indexing with -p flag after upgrade

 

We're in the process of upgrading to VuFind 3.1. Things have gone well, 
however, we're not able to index using the -p flag since we've upgraded. 
The following command fails.

./import-marc.sh -p local/import/import_ole.properties 
VFJan15Incrementals-2015-01-16T1217.mrc

When we run the above command, this appears in the error message:

ERROR [main] (IndexDriver.java:333) - title_full =  custom, 
getAllAlphaExcept(245c)
ERROR [main] (IndexDriver.java:335) - title_full : Unknown extractor 
method: null.getAllAlphaExcept[interface org.marc4j.marc.Record, class 
java.lang.String].

Does anyone have an idea what might be wrong here?

Your help is greatly appreciated.

- Brad


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, <a href="x-msg://2/redir.aspx?REF=iBvdn2AVk_B7iLxpxDsxfkd6YmZpxaC-vYSp3sEozqPllhuQkDXUCAFodHRwczovL25hMDEuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwJTNBJTJGJTJGc2xhc2hkb3Qub3JnJTJGJmRhdGE9MDElN0MwMSU3Q2RlbWlhbi5rYXR6JTQwdmlsbGFub3ZhLmVkdSU3QzQ5YTUwZjQ1OWUwZTRmM2U2NDk3MDhkNDM0ZTg2YTgzJTdDNzY1YThkZTVjZjk0NDRmMDljYWZhZTViZjhjZmEzNjYlN0MxJnNkYXRhPVZlTUM4QW1uY29VeFNZbmVqa202Q09BYk4xSUJTSDlJbW5ERWVxY2k5TlElM0QmcmVzZXJ2ZWQ9MA.." target="_blank" style="color: purple; text-decoration: underline;" class="">SlashDot.org! <a href="x-msg://2/redir.aspx?REF=01OmVpGGSRY6BdLKfCZ_edPfHFIo7JTf8EijYh0uh2fllhuQkDXUCAFodHRwczovL25hMDEuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwJTNBJTJGJTJGc2RtLmxpbmslMkZzbGFzaGRvdCZkYXRhPTAxJTdDMDElN0NkZW1pYW4ua2F0eiU0MHZpbGxhbm92YS5lZHUlN0MyN2M0ODhhNzJlYmE0MTUxMTMzZDA4ZDQyZjU1ZTAwZiU3Qzc2NWE4ZGU1Y2Y5NDQ0ZjA5Y2FmYWU1YmY4Y2ZhMzY2JTdDMSZzZGF0YT0lMkJwTUZOVWU0U05TTXczeWlnNVJ2T3l6dUdCQ1J2YlRteTR1dVJRcldWVmclM0QmcmVzZXJ2ZWQ9MA.." id="LPlnk848716" target="_blank" style="color: purple; text-decoration: underline;" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BpMFNUe4SNSMw3yig5RvOyzuGBCRvbTmy4uuRQrWVVg%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
<a href="x-msg://2/redir.aspx?REF=KKQNPUt5UjlqOP5cVVBTaiawKyE5ojE__QaB2sn6cv3llhuQkDXUCAFtYWlsdG86VnVmaW5kLXRlY2hAbGlzdHMuc291cmNlZm9yZ2UubmV0" target="_blank" style="color: purple; text-decoration: underline;" class="">Vufind-tech@...
<a href="x-msg://2/redir.aspx?REF=PExhAWW-tqDqLwabQrpSg5TurqOFkLxnreGL55i3zGfllhuQkDXUCAFodHRwczovL25hMDEuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwcyUzQSUyRiUyRmxpc3RzLnNvdXJjZWZvcmdlLm5ldCUyRmxpc3RzJTJGbGlzdGluZm8lMkZ2dWZpbmQtdGVjaCZkYXRhPTAxJTdDMDElN0NkZW1pYW4ua2F0eiU0MHZpbGxhbm92YS5lZHUlN0MyN2M0ODhhNzJlYmE0MTUxMTMzZDA4ZDQyZjU1ZTAwZiU3Qzc2NWE4ZGU1Y2Y5NDQ0ZjA5Y2FmYWU1YmY4Y2ZhMzY2JTdDMSZzZGF0YT0lMkJRRFYlMkJlak9kRVVsTTdET3puMG5vMnFCek5yVm9FWjFpJTJGYlBhUWJTYjVnJTNEJnJlc2VydmVkPTA." id="LPlnk271997" target="_blank" style="color: purple; text-decoration: underline;" class="">https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C27c488a72eba4151133d08d42f55e00f%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=%2BQDV%2BejOdEUlM7DOzn0no2qBzNrVoEZ1i%2FbPaQbSb5g%3D&reserved=0

 

 

 

-- 
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to<a href="x-msg://2/redir.aspx?REF=-FpQwLV2aB4kVCk5IkjxPVeCUP79XS5BaVTJwscqgofllhuQkDXUCAFtYWlsdG86c29scm1hcmMtdGVjaCt1bnN1YnNjcmliZUBnb29nbGVncm91cHMuY29t" target="_blank" style="color: purple; text-decoration: underline;" class="">solrmarc-tech+unsubscribe@....
To post to this group, send email to <a href="x-msg://2/redir.aspx?REF=9LjZ-MeaQf5LCnL_ECPHxM3-zk9VOQNNKHVF9_6u0xnllhuQkDXUCAFtYWlsdG86c29scm1hcmMtdGVjaEBnb29nbGVncm91cHMuY29t" target="_blank" style="color: purple; text-decoration: underline;" class="">solrmarc-tech@....
Visit this group at <a href="x-msg://2/redir.aspx?REF=kxPlvA4lO1P6Hako-2wfyatA6ULnm0bI3L_qEB4IOJbllhuQkDXUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2dyb3VwL3NvbHJtYXJjLXRlY2g." target="_blank" style="color: purple; text-decoration: underline;" class="">https://groups.google.com/group/solrmarc-tech.
For more options, visit <a href="x-msg://2/redir.aspx?REF=NzDvr-Bek1erRYwR6Lv2ld_GISJXHevACHIkxlrBl5HllhuQkDXUCAFodHRwczovL2dyb3Vwcy5nb29nbGUuY29tL2Qvb3B0b3V0" target="_blank" style="color: purple; text-decoration: underline;" class="">https://groups.google.com/d/optout.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech