Mapping MARC fields to Indexes

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Mapping MARC fields to Indexes

Greg Pendlebury
Mapping MARC fields to Indexes

Hi Everyone,

We are in the process of exploring replacements for the discovery layer of our ILMS and I'm currently doing an evaluation of vufind's underlying data structure.

I was wondering if anyone could suggest potential areas we should focus our attention on in the data layer because of the setup we currently have.

Our ILMS (Virtua) stores it's MARC data in database BLOBs. We have an Oracle stored procedure (run by and update trigger on the bib table) that passes these blobs into a table whilst retaining it's 'MARCness' ie. tags, subfields, indicators.

This data is already of use to us because we can use our reporting suite on the table for looking at backend reporting.

In terms of making this data accessible in the discovery layer our thinking at this stage is the simplest method would be to run up our own SOLR system around that data, mapping MARC into metadata fields and point vufind at it.

Add in some database triggers for live status/holdings/patron changes and the ILS driver would almost write itself. Our last sticking point would be request/renewals where we will have to some to the hard decision of either faking the database activity our current OPAC generates (tricky during version upgrades), or retaining the OPAC as a backend interface under vufind (continue licensing the OPAC software).

Am I being over-confident or missing a critical link?

Any opinions appreciated.

Ta,

Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841


This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Mapping MARC fields to Indexes

Naomi Dushay
Greg,

Take a look at the solrmarc project -- you can see how both vufind and blacklight map MARC to lucene fields.   Many of us customize our imports further, but it's a great start.  Look especially at vufind.properties and blacklight.properties.


- Naomi

On Oct 9, 2008, at 8:05 PM, Greg Pendlebury wrote:

Hi Everyone,

We are in the process of exploring replacements for the discovery layer of our ILMS and I'm currently doing an evaluation of vufind's underlying data structure.

I was wondering if anyone could suggest potential areas we should focus our attention on in the data layer because of the setup we currently have.

Our ILMS (Virtua) stores it's MARC data in database BLOBs. We have an Oracle stored procedure (run by and update trigger on the bib table) that passes these blobs into a table whilst retaining it's 'MARCness' ie. tags, subfields, indicators.

This data is already of use to us because we can use our reporting suite on the table for looking at backend reporting.

In terms of making this data accessible in the discovery layer our thinking at this stage is the simplest method would be to run up our own SOLR system around that data, mapping MARC into metadata fields and point vufind at it.

Add in some database triggers for live status/holdings/patron changes and the ILS driver would almost write itself. Our last sticking point would be request/renewals where we will have to some to the hard decision of either faking the database activity our current OPAC generates (tricky during version upgrades), or retaining the OPAC as a backend interface under vufind (continue licensing the OPAC software).

Am I being over-confident or missing a critical link?

Any opinions appreciated.

Ta,

Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841


This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech

Naomi Dushay




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Mapping MARC fields to Indexes

wsgrah
Administrator
Greg,

Are you wanting to replace your ILS with Vufind? It's not designed for
that, rather its an alternative front-end to vendor-provided web
interfaces. Basically what it does is provide a front-end that is easily
customized and uses both Solr (for indexing and searching) and your ILS
backend (for things like status, location, etc) in tandem.

I would shy away from actually writing your own ILS...they're
exceptionally complex. If you'd like to take a look at a couple of Open
Source ones that you could easily customize to your liking: Koha
(http://www.koha.org/) and Evergreen (http://open-ils.org/). There is a
Koha driver built for Vufind, and a stub for Koha. There's not a Virtua
driver written yet, but it's really just a matter of writing the correct
SQL query.

If you're wanting to expose this data as an API, you may also want to
take a look at the report developed by the ILS Discovery Interface task
force from the DLF (http://diglib.org/architectures/ilsdi/). It provides
at least some basic guidance as to what several vendors have tacitly
agreed to support (at least at level one).

But as Naomi pointed out, solrmarc will go a long way in getting you
where you want to go in generating the Solr index...the only hard part
is  developing an appropriate schema and their mappings :)

Wayne

Naomi Dushay wrote:

> Greg,
>
> Take a look at the solrmarc project -- you can see how both vufind and
> blacklight map MARC to lucene fields.   Many of us customize our
> imports further, but it's a great start.  Look especially at
> vufind.properties and blacklight.properties.
>
> http://code.google.com/p/solrmarc/
>
> - Naomi
>
> On Oct 9, 2008, at 8:05 PM, Greg Pendlebury wrote:
>
>> Hi Everyone,
>>
>> We are in the process of exploring replacements for the discovery
>> layer of our ILMS and I'm currently doing an evaluation of vufind's
>> underlying data structure.
>>
>> I was wondering if anyone could suggest potential areas we should
>> focus our attention on in the data layer because of the setup we
>> currently have.
>>
>> Our ILMS (Virtua) stores it's MARC data in database BLOBs. We have an
>> Oracle stored procedure (run by and update trigger on the bib table)
>> that passes these blobs into a table whilst retaining it's 'MARCness'
>> ie. tags, subfields, indicators.
>>
>> This data is already of use to us because we can use our reporting
>> suite on the table for looking at backend reporting.
>>
>> In terms of making this data accessible in the discovery layer our
>> thinking at this stage is the simplest method would be to run up our
>> own SOLR system around that data, mapping MARC into metadata fields
>> and point vufind at it.
>>
>> Add in some database triggers for live status/holdings/patron changes
>> and the ILS driver would almost write itself. Our last sticking point
>> would be request/renewals where we will have to some to the hard
>> decision of either faking the database activity our current OPAC
>> generates (tricky during version upgrades), or retaining the OPAC as
>> a backend interface under vufind (continue licensing the OPAC software).
>>
>> Am I being over-confident or missing a critical link?
>>
>> Any opinions appreciated.
>>
>> Ta,
>>
>> *Greg Pendlebury*
>> Electronic Services Officer (Systems Team)
>> Division of Academic Information Services
>> University of Southern Queensland
>> Phone: +61 7 4631 1501
>> Fax: +61 7 4631 1841
>>
>> ------------------------------------------------------------------------
>> This email (including any attached files) is confidential and is for
>> the intended recipient(s) only. If you received this email by
>> mistake, please, as a courtesy, tell the sender, then delete this email.
>>
>> The views and opinions are the originator's and do not necessarily
>> reflect those of the University of Southern Queensland. Although all
>> reasonable precautions were taken to ensure that this email contained
>> no viruses at the time it was sent we accept no liability for any
>> losses arising from its receipt.
>>
>> The University of Southern Queensland is a registered provider of
>> education with the Australian Government (CRICOS Institution Code
>> No's. QLD 00244B / NSW 02225M)
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
>> great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ 
>> <http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________>
>> Vufind-tech mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>
> Naomi Dushay
> [hidden email] <mailto:[hidden email]>
>
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> ------------------------------------------------------------------------
>
> _______________________________________________
> Vufind-tech mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>  

--
/**
  * Wayne Graham
  * Earl Gregg Swem Library
  * PO Box 8794
  * Williamsburg, VA 23188
  * 757.221.3112
  * http://swem.wm.edu/blogs/waynegraham/
  * http://www.liquidfoot.com
  */



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Mapping MARC fields to Indexes

Greg Pendlebury
Thank-you both.

I'll take a closer look at solrmarc and that report.

And no, we aren't replacing the ILMS, just the discovery layer.

Ta,

Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841

-----Original Message-----
From: Wayne Graham [mailto:[hidden email]]
Sent: Saturday, 11 October 2008 3:36 AM
To: Naomi Dushay
Cc: vuFind-Tech
Subject: Re: [VuFind-Tech] Mapping MARC fields to Indexes

Greg,

Are you wanting to replace your ILS with Vufind? It's not designed for
that, rather its an alternative front-end to vendor-provided web
interfaces. Basically what it does is provide a front-end that is easily
customized and uses both Solr (for indexing and searching) and your ILS
backend (for things like status, location, etc) in tandem.

I would shy away from actually writing your own ILS...they're
exceptionally complex. If you'd like to take a look at a couple of Open
Source ones that you could easily customize to your liking: Koha
(http://www.koha.org/) and Evergreen (http://open-ils.org/). There is a
Koha driver built for Vufind, and a stub for Koha. There's not a Virtua
driver written yet, but it's really just a matter of writing the correct
SQL query.

If you're wanting to expose this data as an API, you may also want to
take a look at the report developed by the ILS Discovery Interface task
force from the DLF (http://diglib.org/architectures/ilsdi/). It provides
at least some basic guidance as to what several vendors have tacitly
agreed to support (at least at level one).

But as Naomi pointed out, solrmarc will go a long way in getting you
where you want to go in generating the Solr index...the only hard part
is  developing an appropriate schema and their mappings :)

Wayne

Naomi Dushay wrote:
> Greg,
>
> Take a look at the solrmarc project -- you can see how both vufind and

> blacklight map MARC to lucene fields.   Many of us customize our
> imports further, but it's a great start.  Look especially at
> vufind.properties and blacklight.properties.
>
> http://code.google.com/p/solrmarc/
>
> - Naomi
>
> On Oct 9, 2008, at 8:05 PM, Greg Pendlebury wrote:
>
>> Hi Everyone,
>>
>> We are in the process of exploring replacements for the discovery
>> layer of our ILMS and I'm currently doing an evaluation of vufind's
>> underlying data structure.
>>
>> I was wondering if anyone could suggest potential areas we should
>> focus our attention on in the data layer because of the setup we
>> currently have.
>>
>> Our ILMS (Virtua) stores it's MARC data in database BLOBs. We have an

>> Oracle stored procedure (run by and update trigger on the bib table)
>> that passes these blobs into a table whilst retaining it's 'MARCness'
>> ie. tags, subfields, indicators.
>>
>> This data is already of use to us because we can use our reporting
>> suite on the table for looking at backend reporting.
>>
>> In terms of making this data accessible in the discovery layer our
>> thinking at this stage is the simplest method would be to run up our
>> own SOLR system around that data, mapping MARC into metadata fields
>> and point vufind at it.
>>
>> Add in some database triggers for live status/holdings/patron changes

>> and the ILS driver would almost write itself. Our last sticking point

>> would be request/renewals where we will have to some to the hard
>> decision of either faking the database activity our current OPAC
>> generates (tricky during version upgrades), or retaining the OPAC as
>> a backend interface under vufind (continue licensing the OPAC
software).

>>
>> Am I being over-confident or missing a critical link?
>>
>> Any opinions appreciated.
>>
>> Ta,
>>
>> *Greg Pendlebury*
>> Electronic Services Officer (Systems Team) Division of Academic
>> Information Services University of Southern Queensland
>> Phone: +61 7 4631 1501
>> Fax: +61 7 4631 1841
>>
>> ---------------------------------------------------------------------
>> --- This email (including any attached files) is confidential and is
>> for the intended recipient(s) only. If you received this email by
>> mistake, please, as a courtesy, tell the sender, then delete this
email.
>>
>> The views and opinions are the originator's and do not necessarily
>> reflect those of the University of Southern Queensland. Although all
>> reasonable precautions were taken to ensure that this email contained

>> no viruses at the time it was sent we accept no liability for any
>> losses arising from its receipt.
>>
>> The University of Southern Queensland is a registered provider of
>> education with the Australian Government (CRICOS Institution Code
>> No's. QLD 00244B / NSW 02225M)
>> ---------------------------------------------------------------------
>> ---- This SF.Net email is sponsored by the Moblin Your Move
>> Developer's challenge Build the coolest Linux based applications with

>> Moblin SDK & win great prizes Grand prize is a trip for two to an
>> Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/___________
>> ____________________________________
>> <http://moblin-contest.org/redirect.php?banner_id=100&url=/__________
>> _____________________________________>
>> Vufind-tech mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>
> Naomi Dushay
> [hidden email] <mailto:[hidden email]>
>
>
>
> ----------------------------------------------------------------------
> --
>
> ----------------------------------------------------------------------
> --- This SF.Net email is sponsored by the Moblin Your Move Developer's

> challenge Build the coolest Linux based applications with Moblin SDK &

> win great prizes Grand prize is a trip for two to an Open Source event

> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> ----------------------------------------------------------------------
> --
>
> _______________________________________________
> Vufind-tech mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>  

--
/**
  * Wayne Graham
  * Earl Gregg Swem Library
  * PO Box 8794
  * Williamsburg, VA 23188
  * 757.221.3112
  * http://swem.wm.edu/blogs/waynegraham/
  * http://www.liquidfoot.com
  */



------------------------------------------------------------------------
-
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech

This email (including any attached files) is confidential and is for the
intended recipient(s) only.  If you received this email by mistake,
please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland.  Although all
reasonable precautions were taken to ensure that this email contained no
viruses at the time it was sent we accept no liability for any losses
arising from its receipt.

The University of Southern Queensland is a registered provider of
education with the Australian Government (CRICOS Institution Code No's.
QLD 00244B / NSW 02225M)



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech