Sorting by Relevance

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Sorting by Relevance

Shepard, Thomas - 1150 - MITLL

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

[hidden email]

781 981 0370

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Demian Katz

Are the contents of the indexes identical?

 

If you access the raw Solr URL that VuFind uses to perform a search, you can add debugQuery=true as an additional GET parameter in order to retrieve some information about how the query was executed, etc., and this might offer some data you could compare between the inconsistent servers.

 

Let me know if you need more details!

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:[hidden email]]
Sent: Monday, December 19, 2016 10:02 AM
To: [hidden email]
Subject: [VuFind-General] Sorting by Relevance

 

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

[hidden email]

781 981 0370

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Shepard, Thomas - 1150 - MITLL

Ah, actually the problem/issue seems to occur only when no search term is chosen.

That is, searching without any entered values results in an apparently random sort order (which is different for each install for some reason).

This random sorting also occurs when one browses through the facets.

 

But when I actually enter a search term, I do get a uniform sort across all three installs.

 

So I suppose this is a feature not a bug, right?

 

Thom

 

From: Demian Katz [mailto:[hidden email]]
Sent: Monday, December 19, 2016 11:20 AM
To: Shepard, Thomas - 0050 - MITLL; [hidden email]
Subject: RE: Sorting by Relevance

 

Are the contents of the indexes identical?

 

If you access the raw Solr URL that VuFind uses to perform a search, you can add debugQuery=true as an additional GET parameter in order to retrieve some information about how the query was executed, etc., and this might offer some data you could compare between the inconsistent servers.

 

Let me know if you need more details!

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [[hidden email]]
Sent: Monday, December 19, 2016 10:02 AM
To: [hidden email]
Subject: [VuFind-General] Sorting by Relevance

 

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

[hidden email]

781 981 0370

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Demian Katz

If you indexed your records in a different order on each server, I think that could explain the inconsistency you are seeing. With no other factors to go on, Solr will pick an arbitrary order, which I believe may be influenced by record creation sequence.

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:[hidden email]]
Sent: Monday, December 19, 2016 11:28 AM
To: Demian Katz; [hidden email]
Subject: RE: Sorting by Relevance

 

Ah, actually the problem/issue seems to occur only when no search term is chosen.

That is, searching without any entered values results in an apparently random sort order (which is different for each install for some reason).

This random sorting also occurs when one browses through the facets.

 

But when I actually enter a search term, I do get a uniform sort across all three installs.

 

So I suppose this is a feature not a bug, right?

 

Thom

 

From: Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 11:20 AM
To: Shepard, Thomas - 0050 - MITLL; [hidden email]
Subject: RE: Sorting by Relevance

 

Are the contents of the indexes identical?

 

If you access the raw Solr URL that VuFind uses to perform a search, you can add debugQuery=true as an additional GET parameter in order to retrieve some information about how the query was executed, etc., and this might offer some data you could compare between the inconsistent servers.

 

Let me know if you need more details!

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [[hidden email]]
Sent: Monday, December 19, 2016 10:02 AM
To: [hidden email]
Subject: [VuFind-General] Sorting by Relevance

 

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

[hidden email]

781 981 0370

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Robert Haschart
Demian is right that if none of the specified sort values are different, the records will be ordered based on some default ordering which IIRC is most recently indexed record first.  

However, even if you have re-indexed from scratch into the two separate indexes, using the same set of records, in the same order, with the way that the new SolrMarc uses multiple threads to send chunks of records to the Solr index, the resulting default ordering will be different for the two indexes based on timing differences in the ChunkIndexer threads. 

If it is important that the two indexes have the same default ordering, you could disable the use of multiple threads in the SolrMarc program, but then much of the dramatic speed-up seen with the new version would go away.

-Bob Haschart  (from the SolrMarc project)


From: Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 11:31 AM
To: Shepard, Thomas - 0050 - MITLL; [hidden email]
Subject: Re: [VuFind-General] Sorting by Relevance

If you indexed your records in a different order on each server, I think that could explain the inconsistency you are seeing. With no other factors to go on, Solr will pick an arbitrary order, which I believe may be influenced by record creation sequence.

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:[hidden email]]
Sent: Monday, December 19, 2016 11:28 AM
To: Demian Katz; [hidden email]
Subject: RE: Sorting by Relevance

 

Ah, actually the problem/issue seems to occur only when no search term is chosen.

That is, searching without any entered values results in an apparently random sort order (which is different for each install for some reason).

This random sorting also occurs when one browses through the facets.

 

But when I actually enter a search term, I do get a uniform sort across all three installs.

 

So I suppose this is a feature not a bug, right?

 

Thom

 

From: Demian Katz [mailto:demian.katz@...]
Sent: Monday, December 19, 2016 11:20 AM
To: Shepard, Thomas - 0050 - MITLL; vufind-general@...
Subject: RE: Sorting by Relevance

 

Are the contents of the indexes identical?

 

If you access the raw Solr URL that VuFind uses to perform a search, you can add debugQuery=true as an additional GET parameter in order to retrieve some information about how the query was executed, etc., and this might offer some data you could compare between the inconsistent servers.

 

Let me know if you need more details!

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:tshepard@...]
Sent: Monday, December 19, 2016 10:02 AM
To: vufind-general@...
Subject: [VuFind-General] Sorting by Relevance

 

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

tshepard@...

781 981 0370

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Robert Haschart
Or another possibility that just occurred to me is to add another sorting parameter to the solr call, the subsequent sort specifications only come into play when prior sort orderings produce the same result.  

-Bob


From: Haschart, Robert J. (rh9ec) [[hidden email]]
Sent: Monday, December 19, 2016 12:14 PM
To: Demian Katz; Shepard, Thomas - 0050 - MITLL; [hidden email]
Subject: Re: [VuFind-General] Sorting by Relevance

Demian is right that if none of the specified sort values are different, the records will be ordered based on some default ordering which IIRC is most recently indexed record first.  

However, even if you have re-indexed from scratch into the two separate indexes, using the same set of records, in the same order, with the way that the new SolrMarc uses multiple threads to send chunks of records to the Solr index, the resulting default ordering will be different for the two indexes based on timing differences in the ChunkIndexer threads. 

If it is important that the two indexes have the same default ordering, you could disable the use of multiple threads in the SolrMarc program, but then much of the dramatic speed-up seen with the new version would go away.

-Bob Haschart  (from the SolrMarc project)


From: Demian Katz [[hidden email]]
Sent: Monday, December 19, 2016 11:31 AM
To: Shepard, Thomas - 0050 - MITLL; [hidden email]
Subject: Re: [VuFind-General] Sorting by Relevance

If you indexed your records in a different order on each server, I think that could explain the inconsistency you are seeing. With no other factors to go on, Solr will pick an arbitrary order, which I believe may be influenced by record creation sequence.

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:[hidden email]]
Sent: Monday, December 19, 2016 11:28 AM
To: Demian Katz; [hidden email]
Subject: RE: Sorting by Relevance

 

Ah, actually the problem/issue seems to occur only when no search term is chosen.

That is, searching without any entered values results in an apparently random sort order (which is different for each install for some reason).

This random sorting also occurs when one browses through the facets.

 

But when I actually enter a search term, I do get a uniform sort across all three installs.

 

So I suppose this is a feature not a bug, right?

 

Thom

 

From: Demian Katz [mailto:demian.katz@...]
Sent: Monday, December 19, 2016 11:20 AM
To: Shepard, Thomas - 0050 - MITLL; vufind-general@...
Subject: RE: Sorting by Relevance

 

Are the contents of the indexes identical?

 

If you access the raw Solr URL that VuFind uses to perform a search, you can add debugQuery=true as an additional GET parameter in order to retrieve some information about how the query was executed, etc., and this might offer some data you could compare between the inconsistent servers.

 

Let me know if you need more details!

 

- Demian

 

From: Shepard, Thomas - 0050 - MITLL [mailto:tshepard@...]
Sent: Monday, December 19, 2016 10:02 AM
To: vufind-general@...
Subject: [VuFind-General] Sorting by Relevance

 

I am puzzled and a little concerned that we get three different sort results on our three Vufind installations (production, development, sandbox) when we choose sort by Relevance.

I can understand why our sandbox installation might be different – because it is an earlier version – but our production and development versions are the same, as is searchspecs.yaml.

I’ve also compared searches.ini. Both installations have the following:

 

[Sorting]

relevance = sort_relevance

year = sort_year

year asc = "sort_year asc"

 

Is there an explanation for the discrepancy? Are there other files I need to examine?

 

Thanks,

Thom

 

 

Thom Shepard

MIT Lincoln Lab
244 Wood St.

Lexington, MA 01523

tshepard@...

781 981 0370

 


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general
Reply | Threaded
Open this post in threaded view
|

Re: Sorting by Relevance

Ere Maijala
In reply to this post by Shepard, Thomas - 1150 - MITLL
There's a empty_search_relevance_override setting in searches.ini. It's
commented out by default (I don't quite remember why), but you can use
it to make VuFind use another sort option for an empty search when
sorting by relevancy is selected.

--Ere

19.12.2016, 18.28, Shepard, Thomas - 0050 - MITLL kirjoitti:

> Ah, actually the problem/issue seems to occur only when no search term
> is chosen.
>
> That is, searching without any entered values results in an apparently
> random sort order (which is different for each install for some reason).
>
> This random sorting also occurs when one browses through the facets.
>
>
>
> But when I actually enter a search term, I do get a uniform sort across
> all three installs.
>
>
>
> So I suppose this is a feature not a bug, right?
>
>
>
> Thom
>
>
>
> *From:*Demian Katz [mailto:[hidden email]]
> *Sent:* Monday, December 19, 2016 11:20 AM
> *To:* Shepard, Thomas - 0050 - MITLL; [hidden email]
> *Subject:* RE: Sorting by Relevance
>
>
>
> Are the contents of the indexes identical?
>
>
>
> If you access the raw Solr URL that VuFind uses to perform a search, you
> can add debugQuery=true as an additional GET parameter in order to
> retrieve some information about how the query was executed, etc., and
> this might offer some data you could compare between the inconsistent
> servers.
>
>
>
> Let me know if you need more details!
>
>
>
> - Demian
>
>
>
> *From:* Shepard, Thomas - 0050 - MITLL [mailto:[hidden email]]
> *Sent:* Monday, December 19, 2016 10:02 AM
> *To:* [hidden email]
> <mailto:[hidden email]>
> *Subject:* [VuFind-General] Sorting by Relevance
>
>
>
> I am puzzled and a little concerned that we get three different sort
> results on our three Vufind installations (production, development,
> sandbox) when we choose sort by Relevance.
>
> I can understand why our sandbox installation might be different –
> because it is an earlier version – but our production and development
> versions are the same, as is searchspecs.yaml.
>
> I’ve also compared searches.ini. Both installations have the following:
>
>
>
> [Sorting]
>
> relevance = sort_relevance
>
> year = sort_year
>
> year asc = "sort_year asc"
>
>
>
> Is there an explanation for the discrepancy? Are there other files I
> need to examine?
>
>
>
> Thanks,
>
> Thom
>
>
>
>
>
> Thom Shepard
>
> MIT Lincoln Lab
> 244 Wood St.
>
> Lexington, MA 01523
>
> [hidden email] <mailto:[hidden email]>
>
> 781 981 0370
>
>
>
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
>
>
>
> _______________________________________________
> VuFind-General mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/vufind-general
>

--
Ere Maijala
Kansalliskirjasto / The National Library of Finland

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
VuFind-General mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-general