Adding my custom fields to Solr...

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding my custom fields to Solr...

Layale Bassil

Hello,

 

I am trying to import my records from DSpace into VuFind. I need to know how can I import records with custom fields. Was reading this: https://vufind.org/wiki/development:architecture:solr_index_schema and learnt about the Dynamic fields, how can I use them? Just add any field to my xml file (which will be imported), for example my field is called “dc:interviewer”, just add a field interviewer_str (of type string) to the file and Solr will directly add it to the schema? How can I map it?

 

Please elaborate.

 

Thank you very much.

Regards.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Uwe Reh
Hi,

the default index definition file of VuFind, (schema.xml,
https://github.com/vufind-org/vufind/blob/master/solr/vufind/biblio/conf/schema.xml)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true" stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field
name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this
single valued field. Maybe you would prefer to use "interviewer_str_mv",
which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax,
but it will be sufficient to add something like the next lines in your
'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching
the same pattern. E.g. "interviewer_str", "interviewer_2_str" or
"pet_str" are different but they will be created with the same
attributes. The system of dynamic fields is very powerful and allows
you, to have thousands of new fields just by using them and without the
need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:

> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to know
> how can I import records with custom fields. Was reading this:
> https://vufind.org/wiki/development:architecture:solr_index_schema and
> learnt about the Dynamic fields, how can I use them? Just add any field
> to my xml file (which will be imported), for example my field is called
> “dc:interviewer”, just add a field interviewer_str (of type string) to
> the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Layale Bassil
Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [mailto:[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://github.com/vufind-org/vufind/blob/master/solr/vufind/biblio/conf/schema.xml)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:

> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://vufind.org/wiki/development:architecture:solr_index_schema and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Demian Katz

This wiki page should take you through the process:


https://vufind.org/wiki/development:howtos:displaying_a_custom_field


Note that the templates for displaying fields change dramatically between version 3.x and the upcoming 4.0 release -- depending on which code you are using, the approach is a bit different, and if you upgrade from 3.x to 4.x in the future, you'll need to revisit this. The new approach is a little more abstract than the old approach, but it results in more concise code.


Regardless of which version you are using, please let me know if you still have any questions that the wiki fails to answer.


- Demian




From: Layale Bassil <[hidden email]>
Sent: Tuesday, May 9, 2017 4:58 AM
To: Uwe Reh; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...
 
Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvufind-org%2Fvufind%2Fblob%2Fmaster%2Fsolr%2Fvufind%2Fbiblio%2Fconf%2Fschema.xml&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305212664&sdata=3KKaJpQVhr%2BclKEZJ0P6vB4KjQy6neygiUfzxXWnFrY%3D&reserved=0)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:
> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvufind.org%2Fwiki%2Fdevelopment%3Aarchitecture%3Asolr_index_schema&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=cpcBl7RHlPMuM2At%2Bt2aqSX7h%2B%2BRKI7X1GHk0N5HbAM%3D&reserved=0 and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0 _______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0
_______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Layale Bassil

Hello Demian,

 

Thank you so much. I was able to display custom fields as instructed.

 

I just noticed that OAI-Harvester is not returning the collection, it only returns the records. As I mentioned earlier, I need to use VuFind and be able to preserve the hierarchy of collection/items. First how can I import collections from DSpace to VuFind?

 

Do I have to add the collection manually to Solr? After being able to add collections I will work on the harvested records data in order to be able to show the hierarchy as instructed here: https://vufind.org/wiki/indexing:hierarchies_and_collections

 

Thank you.

 

From: Demian Katz [mailto:[hidden email]]
Sent: Tuesday, May 9, 2017 3:21 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

This wiki page should take you through the process:

 

https://vufind.org/wiki/development:howtos:displaying_a_custom_field

 

Note that the templates for displaying fields change dramatically between version 3.x and the upcoming 4.0 release -- depending on which code you are using, the approach is a bit different, and if you upgrade from 3.x to 4.x in the future, you'll need to revisit this. The new approach is a little more abstract than the old approach, but it results in more concise code.

 

Regardless of which version you are using, please let me know if you still have any questions that the wiki fails to answer.

 

- Demian

 


From: Layale Bassil <[hidden email]>
Sent: Tuesday, May 9, 2017 4:58 AM
To: Uwe Reh; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvufind-org%2Fvufind%2Fblob%2Fmaster%2Fsolr%2Fvufind%2Fbiblio%2Fconf%2Fschema.xml&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305212664&sdata=3KKaJpQVhr%2BclKEZJ0P6vB4KjQy6neygiUfzxXWnFrY%3D&reserved=0)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:
> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvufind.org%2Fwiki%2Fdevelopment%3Aarchitecture%3Asolr_index_schema&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=cpcBl7RHlPMuM2At%2Bt2aqSX7h%2B%2BRKI7X1GHk0N5HbAM%3D&reserved=0 and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0 _______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0
_______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Demian Katz
Great,  I'm glad you were successful.

I'm not sure if there is a way to harvest collections from DSpace. That may be a question worth taking to that community. One unconventional but potentially workable solution might be to alter the XSLT so that it creates collection records at the same time as other records by constructing two Solr add documents at once: one for the collection using the id and title, and the other for the item as usual. The main problem with this idea is that you may have trouble figuring put the collection's parent context unless your hierarchy is very shallow. Otherwise you may be stuck coming up with a different mechanism for indexing the collections.

Let me know if I can help with further brainstorming.

- Demian


From: Layale Bassil <[hidden email]>
Sent: Thursday, May 11, 2017 7:39:38 AM
To: Demian Katz; Uwe Reh; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...
 

Hello Demian,

 

Thank you so much. I was able to display custom fields as instructed.

 

I just noticed that OAI-Harvester is not returning the collection, it only returns the records. As I mentioned earlier, I need to use VuFind and be able to preserve the hierarchy of collection/items. First how can I import collections from DSpace to VuFind?

 

Do I have to add the collection manually to Solr? After being able to add collections I will work on the harvested records data in order to be able to show the hierarchy as instructed here: https://vufind.org/wiki/indexing:hierarchies_and_collections

 

Thank you.

 

From: Demian Katz [mailto:[hidden email]]
Sent: Tuesday, May 9, 2017 3:21 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

This wiki page should take you through the process:

 

https://vufind.org/wiki/development:howtos:displaying_a_custom_field

 

Note that the templates for displaying fields change dramatically between version 3.x and the upcoming 4.0 release -- depending on which code you are using, the approach is a bit different, and if you upgrade from 3.x to 4.x in the future, you'll need to revisit this. The new approach is a little more abstract than the old approach, but it results in more concise code.

 

Regardless of which version you are using, please let me know if you still have any questions that the wiki fails to answer.

 

- Demian

 


From: Layale Bassil <[hidden email]>
Sent: Tuesday, May 9, 2017 4:58 AM
To: Uwe Reh; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvufind-org%2Fvufind%2Fblob%2Fmaster%2Fsolr%2Fvufind%2Fbiblio%2Fconf%2Fschema.xml&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305212664&sdata=3KKaJpQVhr%2BclKEZJ0P6vB4KjQy6neygiUfzxXWnFrY%3D&reserved=0)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:
> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvufind.org%2Fwiki%2Fdevelopment%3Aarchitecture%3Asolr_index_schema&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=cpcBl7RHlPMuM2At%2Bt2aqSX7h%2B%2BRKI7X1GHk0N5HbAM%3D&reserved=0 and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0 _______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0
_______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Layale Bassil

Hello Demian,

 

What do you mean by a very shallow hierarchy?

 

Second, which xslt file do I have to edit? Any guidance in here?

 

Do you think it is better to manually create xml files (similar to the ones harvested from DSpace) for the collections and import them to Solr? Then try to take the ID of the collection as inserted and use it in the xml files of the records that belong to that collection? Do you think this might work?

 

Please advise.

 

Thank you again,

Regards.

 

From: Demian Katz [mailto:[hidden email]]
Sent: Thursday, May 11, 2017 3:54 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...

 

Great,  I'm glad you were successful.

I'm not sure if there is a way to harvest collections from DSpace. That may be a question worth taking to that community. One unconventional but potentially workable solution might be to alter the XSLT so that it creates collection records at the same time as other records by constructing two Solr add documents at once: one for the collection using the id and title, and the other for the item as usual. The main problem with this idea is that you may have trouble figuring put the collection's parent context unless your hierarchy is very shallow. Otherwise you may be stuck coming up with a different mechanism for indexing the collections.

Let me know if I can help with further brainstorming.

- Demian


From: Layale Bassil <[hidden email]>
Sent: Thursday, May 11, 2017 7:39:38 AM
To: Demian Katz; Uwe Reh; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...

 

Hello Demian,

 

Thank you so much. I was able to display custom fields as instructed.

 

I just noticed that OAI-Harvester is not returning the collection, it only returns the records. As I mentioned earlier, I need to use VuFind and be able to preserve the hierarchy of collection/items. First how can I import collections from DSpace to VuFind?

 

Do I have to add the collection manually to Solr? After being able to add collections I will work on the harvested records data in order to be able to show the hierarchy as instructed here: https://vufind.org/wiki/indexing:hierarchies_and_collections

 

Thank you.

 

From: Demian Katz [[hidden email]]
Sent: Tuesday, May 9, 2017 3:21 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

This wiki page should take you through the process:

 

https://vufind.org/wiki/development:howtos:displaying_a_custom_field

 

Note that the templates for displaying fields change dramatically between version 3.x and the upcoming 4.0 release -- depending on which code you are using, the approach is a bit different, and if you upgrade from 3.x to 4.x in the future, you'll need to revisit this. The new approach is a little more abstract than the old approach, but it results in more concise code.

 

Regardless of which version you are using, please let me know if you still have any questions that the wiki fails to answer.

 

- Demian

 


From: Layale Bassil <[hidden email]>
Sent: Tuesday, May 9, 2017 4:58 AM
To: Uwe Reh; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvufind-org%2Fvufind%2Fblob%2Fmaster%2Fsolr%2Fvufind%2Fbiblio%2Fconf%2Fschema.xml&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305212664&sdata=3KKaJpQVhr%2BclKEZJ0P6vB4KjQy6neygiUfzxXWnFrY%3D&reserved=0)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:
> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvufind.org%2Fwiki%2Fdevelopment%3Aarchitecture%3Asolr_index_schema&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=cpcBl7RHlPMuM2At%2Bt2aqSX7h%2B%2BRKI7X1GHk0N5HbAM%3D&reserved=0 and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0 _______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0
_______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Adding my custom fields to Solr...

Demian Katz

When I say a very shallow hierarchy, I basically mean no more than two levels. If the item-level records already include the top-level collection and the immediate parent collection, that allows us to infer the context of the collection if we know that it is either the top-level collection, or has the top-level collection as its immediate parent. If there are more than two levels, then we don't have enough information to build a collection record (unless we also have the "grandparent" record included in the data).


I assume that you have already customized an XSLT file for your DSpace ingest, so I would just suggest adding some more code to that existing file if you were able to go forward with this approach.


If you have a relatively small number of collections, it certainly might be easier to just manually create the XML and post them to Solr directly. Again, my lack of knowledge of DSpace internals limits my suggestions here -- perhaps this could be achieved with a few simple database queries, in which case that's almost certainly the most straightforward approach, since the collection records don't really need to include much detail beyond the title and the relevant hierarchical details.


- Demian




From: Layale Bassil <[hidden email]>
Sent: Friday, May 12, 2017 4:52 AM
To: Demian Katz; Uwe Reh; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...
 

Hello Demian,

 

What do you mean by a very shallow hierarchy?

 

Second, which xslt file do I have to edit? Any guidance in here?

 

Do you think it is better to manually create xml files (similar to the ones harvested from DSpace) for the collections and import them to Solr? Then try to take the ID of the collection as inserted and use it in the xml files of the records that belong to that collection? Do you think this might work?

 

Please advise.

 

Thank you again,

Regards.

 

From: Demian Katz [mailto:[hidden email]]
Sent: Thursday, May 11, 2017 3:54 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...

 

Great,  I'm glad you were successful.

I'm not sure if there is a way to harvest collections from DSpace. That may be a question worth taking to that community. One unconventional but potentially workable solution might be to alter the XSLT so that it creates collection records at the same time as other records by constructing two Solr add documents at once: one for the collection using the id and title, and the other for the item as usual. The main problem with this idea is that you may have trouble figuring put the collection's parent context unless your hierarchy is very shallow. Otherwise you may be stuck coming up with a different mechanism for indexing the collections.

Let me know if I can help with further brainstorming.

- Demian


From: Layale Bassil <[hidden email]>
Sent: Thursday, May 11, 2017 7:39:38 AM
To: Demian Katz; Uwe Reh; [hidden email]
Subject: RE: [VuFind-General] Adding my custom fields to Solr...

 

Hello Demian,

 

Thank you so much. I was able to display custom fields as instructed.

 

I just noticed that OAI-Harvester is not returning the collection, it only returns the records. As I mentioned earlier, I need to use VuFind and be able to preserve the hierarchy of collection/items. First how can I import collections from DSpace to VuFind?

 

Do I have to add the collection manually to Solr? After being able to add collections I will work on the harvested records data in order to be able to show the hierarchy as instructed here: https://vufind.org/wiki/indexing:hierarchies_and_collections

 

Thank you.

 

From: Demian Katz [[hidden email]]
Sent: Tuesday, May 9, 2017 3:21 PM
To: Layale Bassil <[hidden email]>; Uwe Reh <[hidden email]>; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

This wiki page should take you through the process:

 

https://vufind.org/wiki/development:howtos:displaying_a_custom_field

 

Note that the templates for displaying fields change dramatically between version 3.x and the upcoming 4.0 release -- depending on which code you are using, the approach is a bit different, and if you upgrade from 3.x to 4.x in the future, you'll need to revisit this. The new approach is a little more abstract than the old approach, but it results in more concise code.

 

Regardless of which version you are using, please let me know if you still have any questions that the wiki fails to answer.

 

- Demian

 


From: Layale Bassil <[hidden email]>
Sent: Tuesday, May 9, 2017 4:58 AM
To: Uwe Reh; [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

 

Thank you very much Uwe, it worked!

I need to know now how can I display this field in VuFind? I want to add it under Author in the normal view of the record in VuFind... which files should I edit?

Many thanks again!
Regards.

-----Original Message-----
From: Uwe Reh [[hidden email]]
Sent: Monday, May 8, 2017 8:08 PM
To: [hidden email]
Subject: Re: [VuFind-General] Adding my custom fields to Solr...

Hi,

the default index definition file of VuFind, (schema.xml,
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvufind-org%2Fvufind%2Fblob%2Fmaster%2Fsolr%2Fvufind%2Fbiblio%2Fconf%2Fschema.xml&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305212664&sdata=3KKaJpQVhr%2BclKEZJ0P6vB4KjQy6neygiUfzxXWnFrY%3D&reserved=0)
contains two definitions for dynamic created string fields.
> <dynamicField name="*_str"    type="string" indexed="true" stored="true"/>
> <dynamicField name="*_str_mv" type="string" indexed="true"
> stored="true" multiValued="true"/>
This lines allow SOLR to create automatically new fields for any field name which matches the pattern.

Eg. the first usage of "interviewer_str" will automatically create this single valued field. Maybe you would prefer to use "interviewer_str_mv", which allows to have more than one interviewer for one document.

Since I've never used the xml import, I'm not sure about the the syntax, but it will be sufficient to add something like the next lines in your 'xsl' file.
> <xsl:if test="//dc:interviewer">
>    <xsl:for-each select="//dc:interviewer">
>      <field name="interviewer_str_mv"><xsl:value-of select="normalize-space()"/></field>
>   </xsl:for-each>
> </xsl:if>

Additional to "interviewer_str" you can also have more fields matching the same pattern. E.g. "interviewer_str", "interviewer_2_str" or "pet_str" are different but they will be created with the same attributes. The system of dynamic fields is very powerful and allows you, to have thousands of new fields just by using them and without the need, to change the schema.xml file.

Regards
Uwe




Am 08.05.2017 um 16:36 schrieb Layale Bassil:
> Hello,
>
>
>
> I am trying to import my records from DSpace into VuFind. I need to
> know how can I import records with custom fields. Was reading this:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvufind.org%2Fwiki%2Fdevelopment%3Aarchitecture%3Asolr_index_schema&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=cpcBl7RHlPMuM2At%2Bt2aqSX7h%2B%2BRKI7X1GHk0N5HbAM%3D&reserved=0 and
> learnt about the Dynamic fields, how can I use them? Just add any
> field to my xml file (which will be imported), for example my field is
> called "dc:interviewer", just add a field interviewer_str (of type
> string) to the file and Solr will directly add it to the schema? How can I map it?
>
>
>
> Please elaborate.
>
>
>
> Thank you very much.
>
> Regards.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0 _______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=l1frxLROYFXsDFNJeFVc%2FxX86DMs3VWmiZeqhktilio%3D&reserved=0
_______________________________________________
VuFind-General mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-general&data=02%7C01%7Cdemian.katz%40villanova.edu%7C31dc250b45954eaa5d0808d496b9d145%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636299172305222672&sdata=bpQ4nlQeE2q0ZKcNaQrNCjI9wsTvb15%2FReXSDB1haqA%3D&reserved=0


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general