HELP! problem with fullrecord conversion via MARCFlat

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

HELP! problem with fullrecord conversion via MARCFlat

Naomi Dushay
I re-indexed our MARC data over the weekend, and it went swimmingly except Record.php is now unable to parse the 'fullrecord' field into the $record array.

This was not a problem before I re-indexed.  Here's what changed:

1.  I had to generate the index on my mac, instead of on our linux box, and then I ftp-ed them over to the linux box.
2.  I changed the fullrecord field from 
     <field name="fullrecord" type="text" indexed="true" stored="true"/>
to
     <field name="fullrecord" type="string" indexed="false" stored="true"/>
   
I can't imagine how 2. would change anything about how the field is *stored* in the index, but these are the things  I changed.

Looking at MARCFlat.php, it appears to expect each marc field to end with a newline character.  Our data does not seem to have newlines in it.

We can now see #30;  and #31;  and #29;   in the field when we look at the raw solr response in Firefox.  I have no idea if that was present before.  Sadly, due to temporary space constraints, I was unable to keep the old index, so I can't compare our current "fullrecord" field contents with the former contents of the field.

I know that 
#30;  -->  \x1E --> MARC::END_OF_FIELD
#31;  -->  \x1F --> MARC::SUBFIELD_INDICATOR
#29:  -->  \x1D --> MARC::END_OF_RECORD

Jessie and I have been fiddling around with trying to explode on the end-of-field variants, and trying to convert them to newline chars.  We have had no joy.

Can anyone help us?

Thank you,


Naomi Dushay




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Fwd: HELP! problem with fullrecord conversion via MARCFlat

Naomi Dushay
I should have included the contents of the fullrecord field from a raw solr response (via web browser):

<str name="fullrecord">
00775cam a2200289ua 4500001000800000004000900008005001700017008004100034010002000075020002600095020002200121020001800143020002500161035002700186035002100213040001000234050002100244082001600265100002100281245002800302260003200330300003400362440003200396500002000428596000700448650003000455#30;AGY8297#30;a1502056#30;19910712000000.0#30;830513s1983    enka          001 0 eng  #30;  #31;a   83167620 /MN#30;  #31;a0356090965 :#31;c£10.95#30;  #31;a0356090973 (pbk.)#30;  #31;a9780356090962#30;  #31;a9780356090979 (pbk.)#30;  #31;a(CStRLIN)CSUG83-B24298#30;  #31;a(OCoLC-M)9689561#30;  #31;dOrLoB#30;0 #31;aML955#31;b.T82 1983#30;0 #31;a788/.41#31;219#30;1 #31;aTuckwell, Barry.#30;10#31;aHorn /#31;cBarry Tuckwell.#30;  #31;aLondon :#31;bMacdonald,#31;c1983.#30;  #31;axxi, 202 p. :#31;bill. ;#31;c23 cm.#30; 0#31;aYehudi Menuhin music guides#30;  #31;aIncludes index.#30;  #31;a18#30; 0#31;aHorn (Musical instrument)#30;#29;
</str>

Isn't it pretty?
- Naomi

Begin forwarded message:

From: Naomi Dushay <[hidden email]>
Date: August 6, 2008 11:12:06 AM PDT
Subject: HELP!  problem with fullrecord conversion via MARCFlat

I re-indexed our MARC data over the weekend, and it went swimmingly except Record.php is now unable to parse the 'fullrecord' field into the $record array.

This was not a problem before I re-indexed.  Here's what changed:

1.  I had to generate the index on my mac, instead of on our linux box, and then I ftp-ed them over to the linux box.
2.  I changed the fullrecord field from 
     <field name="fullrecord" type="text" indexed="true" stored="true"/>
to
     <field name="fullrecord" type="string" indexed="false" stored="true"/>
   
I can't imagine how 2. would change anything about how the field is *stored* in the index, but these are the things  I changed.

Looking at MARCFlat.php, it appears to expect each marc field to end with a newline character.  Our data does not seem to have newlines in it.

We can now see #30;  and #31;  and #29;   in the field when we look at the raw solr response in Firefox.  I have no idea if that was present before.  Sadly, due to temporary space constraints, I was unable to keep the old index, so I can't compare our current "fullrecord" field contents with the former contents of the field.

I know that 
#30;  -->  \x1E --> MARC::END_OF_FIELD
#31;  -->  \x1F --> MARC::SUBFIELD_INDICATOR
#29:  -->  \x1D --> MARC::END_OF_RECORD

Jessie and I have been fiddling around with trying to explode on the end-of-field variants, and trying to convert them to newline chars.  We have had no joy.

Can anyone help us?

Thank you,


Naomi Dushay




Naomi Dushay




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: HELP! problem with fullrecord conversion via MARCFlat

Andrew Nagy-2
Naomi - I made a change to vufind/web/services/Record/Record.php that will probably break the record view (I didn't mean to check it in).  You might want to try reverting the Record.php to the 2nd latest edition.

Andrew
________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Naomi Dushay [[hidden email]]
Sent: Wednesday, August 06, 2008 4:24 PM
To: [hidden email]
Subject: [VuFind-Tech] Fwd: HELP! problem with fullrecord conversion via        MARCFlat

I should have included the contents of the fullrecord field from a raw solr response (via web browser):

<str name="fullrecord">
00775cam a2200289ua 4500001000800000004000900008005001700017008004100034010002000075020002600095020002200121020001800143020002500161035002700186035002100213040001000234050002100244082001600265100002100281245002800302260003200330300003400362440003200396500002000428596000700448650003000455#30;AGY8297#30;a1502056#30;19910712000000.0#30;830513s1983    enka          001 0 eng  #30;  #31;a   83167620 /MN#30;  #31;a0356090965 :#31;c£10.95#30;  #31;a0356090973 (pbk.)#30;  #31;a9780356090962#30;  #31;a9780356090979 (pbk.)#30;  #31;a(CStRLIN)CSUG83-B24298#30;  #31;a(OCoLC-M)9689561#30;  #31;dOrLoB#30;0 #31;aML955#31;b.T82 1983#30;0 #31;a788/.41#31;219#30;1 #31;aTuckwell, Barry.#30;10#31;aHorn /#31;cBarry Tuckwell.#30;  #31;aLondon :#31;bMacdonald,#31;c1983.#30;  #31;axxi, 202 p. :#31;bill. ;#31;c23 cm.#30; 0#31;aYehudi Menuhin music guides#30;  #31;aIncludes index.#30;  #31;a18#30; 0#31;aHorn (Musical instrument)#30;#29;
</str>

Isn't it pretty?
- Naomi

Begin forwarded message:

From: Naomi Dushay <[hidden email]<mailto:[hidden email]>>
Date: August 6, 2008 11:12:06 AM PDT
To: [hidden email]<mailto:[hidden email]>
Subject: HELP!  problem with fullrecord conversion via MARCFlat

I re-indexed our MARC data over the weekend, and it went swimmingly except Record.php is now unable to parse the 'fullrecord' field into the $record array.

This was not a problem before I re-indexed.  Here's what changed:

1.  I had to generate the index on my mac, instead of on our linux box, and then I ftp-ed them over to the linux box.
2.  I changed the fullrecord field from
     <field name="fullrecord" type="text" indexed="true" stored="true"/>
to
     <field name="fullrecord" type="string" indexed="false" stored="true"/>

I can't imagine how 2. would change anything about how the field is *stored* in the index, but these are the things  I changed.

Looking at MARCFlat.php, it appears to expect each marc field to end with a newline character.  Our data does not seem to have newlines in it.

We can now see #30;  and #31;  and #29;   in the field when we look at the raw solr response in Firefox.  I have no idea if that was present before.  Sadly, due to temporary space constraints, I was unable to keep the old index, so I can't compare our current "fullrecord" field contents with the former contents of the field.

I know that
#30;  -->  \x1E --> MARC::END_OF_FIELD
#31;  -->  \x1F --> MARC::SUBFIELD_INDICATOR
#29:  -->  \x1D --> MARC::END_OF_RECORD

Jessie and I have been fiddling around with trying to explode on the end-of-field variants, and trying to convert them to newline chars.  We have had no joy.

Can anyone help us?

Thank you,


Naomi Dushay
[hidden email]<mailto:[hidden email]>




Naomi Dushay
[hidden email]<mailto:[hidden email]>




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: HELP! problem with fullrecord conversion via MARCFlat

Naomi Dushay
Andrew,

I haven't updated my code recently because of your post about lots of  
unstable development right now.

I am running with Alan Rykhus's theory that running the index job on  
my Mac is the problem.  I am going to regenerate the index this  
weekend on our linux box.  Hopefully the problem will disappear like  
magic!

Thanks to all for your help.  This community is awesome!

- Naomi


On Aug 6, 2008, at 3:58 PM, Andrew Nagy wrote:

> Naomi - I made a change to vufind/web/services/Record/Record.php  
> that will probably break the record view (I didn't mean to check it  
> in).  You might want to try reverting the Record.php to the 2nd  
> latest edition.
>
> Andrew
> ________________________________________
> From: [hidden email] [[hidden email]
> ] On Behalf Of Naomi Dushay [[hidden email]]
> Sent: Wednesday, August 06, 2008 4:24 PM
> To: [hidden email]
> Subject: [VuFind-Tech] Fwd: HELP! problem with fullrecord conversion  
> via        MARCFlat
>
> I should have included the contents of the fullrecord field from a  
> raw solr response (via web browser):
>
> <str name="fullrecord">
> 00775cam a2200289ua  
> 4500001000800000004000900008005001700017008004100034010002000075020002600095020002200121020001800143020002500161035002700186035002100213040001000234050002100244082001600265100002100281245002800302260003200330300003400362440003200396500002000428596000700448650003000455
> #30;AGY8297#30;a1502056#30;19910712000000.0#30;830513s1983    
> enka          001 0 eng  #30;  #31;a   83167620 /MN#30;  
> #31;a0356090965 :#31;c£10.95#30;  #31;a0356090973 (pbk.)#30;  
> #31;a9780356090962#30;  #31;a9780356090979 (pbk.)#30;  
> #31;a(CStRLIN)CSUG83-B24298#30;  #31;a(OCoLC-M)9689561#30;  
> #31;dOrLoB#30;0 #31;aML955#31;b.T82 1983#30;0  
> #31;a788/.41#31;219#30;1 #31;aTuckwell, Barry.#30;10#31;aHorn /
> #31;cBarry Tuckwell.#30;  
> #31;aLondon :#31;bMacdonald,#31;c1983.#30;  #31;axxi, 202  
> p. :#31;bill. ;#31;c23 cm.#30; 0#31;aYehudi Menuhin music  
> guides#30;  #31;aIncludes index.#30;  #31;a18#30; 0#31;aHorn  
> (Musical instrument)#30;#29;
> </str>
>
> Isn't it pretty?
> - Naomi
>
> Begin forwarded message:
>
> From: Naomi Dushay <[hidden email]<mailto:[hidden email]>>
> Date: August 6, 2008 11:12:06 AM PDT
> To: [hidden email]<mailto:[hidden email]
> >
> Subject: HELP!  problem with fullrecord conversion via MARCFlat
>
> I re-indexed our MARC data over the weekend, and it went swimmingly  
> except Record.php is now unable to parse the 'fullrecord' field into  
> the $record array.
>
> This was not a problem before I re-indexed.  Here's what changed:
>
> 1.  I had to generate the index on my mac, instead of on our linux  
> box, and then I ftp-ed them over to the linux box.
> 2.  I changed the fullrecord field from
>     <field name="fullrecord" type="text" indexed="true"  
> stored="true"/>
> to
>     <field name="fullrecord" type="string" indexed="false"  
> stored="true"/>
>
> I can't imagine how 2. would change anything about how the field is  
> *stored* in the index, but these are the things  I changed.
>
> Looking at MARCFlat.php, it appears to expect each marc field to end  
> with a newline character.  Our data does not seem to have newlines  
> in it.
>
> We can now see #30;  and #31;  and #29;   in the field when we look  
> at the raw solr response in Firefox.  I have no idea if that was  
> present before.  Sadly, due to temporary space constraints, I was  
> unable to keep the old index, so I can't compare our current  
> "fullrecord" field contents with the former contents of the field.
>
> I know that
> #30;  -->  \x1E --> MARC::END_OF_FIELD
> #31;  -->  \x1F --> MARC::SUBFIELD_INDICATOR
> #29:  -->  \x1D --> MARC::END_OF_RECORD
>
> Jessie and I have been fiddling around with trying to explode on the  
> end-of-field variants, and trying to convert them to newline chars.  
> We have had no joy.
>
> Can anyone help us?
>
> Thank you,
>
>
> Naomi Dushay
> [hidden email]<mailto:[hidden email]>
>
>
>
>
> Naomi Dushay
> [hidden email]<mailto:[hidden email]>
>
>
>

Naomi Dushay
[hidden email]




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: HELP! problem with fullrecord conversion via MARCFlat

Steve Richardson
In reply to this post by Naomi Dushay

We had the same problem.

 

The marc record stored in solr has changed from a tagged format to  regular marc. We changed 3 files

 

1. Changed the _decode function in MARCFLAT.php to this

 

    private function _decode($text)

    {

 

        $text = trim($text);

        $text = preg_replace('/#31;/', "\x1F", $text);

        $text = preg_replace('/#30;/', "\x1E", $text);

 

        $marcfile = new File_MARC($text, File_MARC::SOURCE_STRING);

        $mymarc = $marcfile->next();

 

        return $mymarc;

    }

 

 

2. Change /vufind/web/services/Record/Record.php to fix the leader

 

From  $this->record['LEADER'] = str_replace('LEADER', '', $marcRecord->getLeader());

To    $this->record['LEADER'] = $marcRecord->getLeader();

 

 

3. And  /vufind/web/services/Record/xsl/record-marc.xsl  Change style sheet to fix leader

 

      <td colspan="3"><xsl:value-of select="marc:leader"/></td>

 

 

Hope this helps.

 

Cheers Steve

 

 

 

 

 

 


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: HELP! problem with fullrecord conversion via MARCFlat

Naomi Dushay
Steve,

Your fix worked like a charm!  I'm ecstatic!  I owe you one.

- Naomi

On Aug 6, 2008, at 5:51 PM, Steve Richardson wrote:

We had the same problem.
 
The marc record stored in solr has changed from a tagged format to  regular marc. We changed 3 files
 
1. Changed the _decode function in MARCFLAT.php to this
 
    private function _decode($text)
    {
 
        $text = trim($text);
        $text = preg_replace('/#31;/', "\x1F", $text);
        $text = preg_replace('/#30;/', "\x1E", $text);
 
        $marcfile = new File_MARC($text, File_MARC::SOURCE_STRING);
        $mymarc = $marcfile->next();
 
        return $mymarc;
    }
 
 
2. Change /vufind/web/services/Record/Record.php to fix the leader
 
From  $this->record['LEADER'] = str_replace('LEADER', '', $marcRecord->getLeader());
To    $this->record['LEADER'] = $marcRecord->getLeader();
 
 
3. And  /vufind/web/services/Record/xsl/record-marc.xsl  Change style sheet to fix leader
 
      <td colspan="3"><xsl:value-of select="marc:leader"/></td>
 
 
Hope this helps.
 
Cheers Steve
 
 
 
 
 
 
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech

Naomi Dushay




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: HELP! problem with fullrecord conversion via MARCFlat

Andrew Nagy-2
In reply to this post by Steve Richardson

Steve - thanks for the patch!  The reason for this is due to the new importer.  The new importer stores the MARC record in the index as binary marc rather than flat text.  I have implemented your changes into the code.

 

Thanks

Andrew

 

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Steve Richardson
Sent: Wednesday, August 06, 2008 8:51 PM
To: [hidden email]
Subject: Re: [VuFind-Tech] HELP! problem with fullrecord conversion via MARCFlat

 

We had the same problem.

 

The marc record stored in solr has changed from a tagged format to  regular marc. We changed 3 files

 

1. Changed the _decode function in MARCFLAT.php to this

 

    private function _decode($text)

    {

 

        $text = trim($text);

        $text = preg_replace('/#31;/', "\x1F", $text);

        $text = preg_replace('/#30;/', "\x1E", $text);

 

        $marcfile = new File_MARC($text, File_MARC::SOURCE_STRING);

        $mymarc = $marcfile->next();

 

        return $mymarc;

    }

 

 

2. Change /vufind/web/services/Record/Record.php to fix the leader

 

From  $this->record['LEADER'] = str_replace('LEADER', '', $marcRecord->getLeader());

To    $this->record['LEADER'] = $marcRecord->getLeader();

 

 

3. And  /vufind/web/services/Record/xsl/record-marc.xsl  Change style sheet to fix leader

 

      <td colspan="3"><xsl:value-of select="marc:leader"/></td>

 

 

Hope this helps.

 

Cheers Steve

 

 

 

 

 

 


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech