Query forwarding (search.ini) for articles@DOAJ

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Query forwarding (search.ini) for articles@DOAJ

P. S. Mukhopadhyay

Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.
--
-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

Demian Katz

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

P. S. Mukhopadhyay
Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--
-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

Demian Katz

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

P. S. Mukhopadhyay
Dear Demian

Thanks for the detail explanation. Yes you are right that the query string is formatted in an unusual way. So i tried this string where query placed at the end directly in the address bar.

<a href="https://doaj.org/search?source={">https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "web ontology"}}}}}

And it worked (retrieved 636 articles).

Inspired by this small success I entered this string in the searchbox.ini

target[] = "<a href="https://doaj.org/search?source={">https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": ""}}}}}"

But the problem is that VuFind query forwarding mechanism placing the search term at the end and not within the braces.

<a href="https://doaj.org/search?source={query">https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {query_string: {default_operator: AND,query: }}}}}web+ontology

I got stuck here....

Regards

On Tue, Mar 21, 2017 at 11:13 PM, Demian Katz <[hidden email]> wrote:

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--
-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

Demian Katz

Yes, this is the key problem – the VuFind code assumes the search term always goes on the end; it was not designed for this unusual case where the search term is actually part of a data structure.

 

Here is the relevant code:

 

https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/Controller/CombinedController.php#L246

 

You could perhaps revise this to something like:

 

        case 'External':

            $lookfor = $this->params()->fromQuery('lookfor');

            if (false === strpos($lookfor, '%%lookfor%%')) {

                return $this->redirect()->toUrl($target . urlencode($lookfor));

            } else {

                return $this->redirect()->toUrl(str_replace('%%lookfor%%', urlencode($lookfor), $target));

            }

 

And then in your target[] setting, you would put %%lookfor%% into the URL at the spot where the query string belongs.

 

Does that make sense? If it helps, we can consider making this a standard feature in a future release.

 

- Demian

 

 

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 2:00 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

Thanks for the detail explanation. Yes you are right that the query string is formatted in an unusual way. So i tried this string where query placed at the end directly in the address bar.

https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "web ontology"}}}}}

And it worked (retrieved 636 articles).

Inspired by this small success I entered this string in the searchbox.ini

target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": ""}}}}}"

But the problem is that VuFind query forwarding mechanism placing the search term at the end and not within the braces.


https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {query_string: {default_operator: AND,query: }}}}}web+ontology

I got stuck here....

Regards

 

On Tue, Mar 21, 2017 at 11:13 PM, Demian Katz <[hidden email]> wrote:

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

P. S. Mukhopadhyay
Yes. Changed code and tried all these three possibilities

;target[] = "<a href="https://doaj.org/search?source={">https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "%%lookfor%%"}}}}}"
;target[] = "<a href="https://doaj.org/search?source={">https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","%%lookfor%%"}}}}}"
target[] = "<a href="https://doaj.org/search?source={">https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"%%lookfor%%"}}}}}"

but still query is going after the braces.

<a href="https://doaj.org/search?source={query">https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {%%lookfor%%}}}}}web

Am I doing some mistake in placing %%lookfor%%

On Tue, Mar 21, 2017 at 11:44 PM, Demian Katz <[hidden email]> wrote:

Yes, this is the key problem – the VuFind code assumes the search term always goes on the end; it was not designed for this unusual case where the search term is actually part of a data structure.

 

Here is the relevant code:

 

https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/Controller/CombinedController.php#L246

 

You could perhaps revise this to something like:

 

        case 'External':

            $lookfor = $this->params()->fromQuery('lookfor');

            if (false === strpos($lookfor, '%%lookfor%%')) {

                return $this->redirect()->toUrl($target . urlencode($lookfor));

            } else {

                return $this->redirect()->toUrl(str_replace('%%lookfor%%', urlencode($lookfor), $target));

            }

 

And then in your target[] setting, you would put %%lookfor%% into the URL at the spot where the query string belongs.

 

Does that make sense? If it helps, we can consider making this a standard feature in a future release.

 

- Demian

 

 

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 2:00 PM


To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

Thanks for the detail explanation. Yes you are right that the query string is formatted in an unusual way. So i tried this string where query placed at the end directly in the address bar.

https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "web ontology"}}}}}

And it worked (retrieved 636 articles).

Inspired by this small success I entered this string in the searchbox.ini

target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": ""}}}}}"

But the problem is that VuFind query forwarding mechanism placing the search term at the end and not within the braces.


https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {query_string: {default_operator: AND,query: }}}}}web+ontology

I got stuck here....

Regards

 

On Tue, Mar 21, 2017 at 11:13 PM, Demian Katz <[hidden email]> wrote:

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--
-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

Demian Katz

It is possible that there is a logical flaw in my code, since I shared it without actually testing it. I will try to find more time to actually try this out in the next few days; unfortunately this week is quite busy, so I apologize if I do not get to this in as timely a fashion as I would like. If you have time to do some troubleshooting and get it working in the meantime, please share any working version you come up with. Putting some die() statements in strategic places in the code to ensure that it is flowing as expected might be helpful.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 3:21 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Yes. Changed code and tried all these three possibilities

;target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "%%lookfor%%"}}}}}"
;target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","%%lookfor%%"}}}}}"
target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"%%lookfor%%"}}}}}"

but still query is going after the braces.

https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {%%lookfor%%}}}}}web

Am I doing some mistake in placing %%lookfor%%

 

On Tue, Mar 21, 2017 at 11:44 PM, Demian Katz <[hidden email]> wrote:

Yes, this is the key problem – the VuFind code assumes the search term always goes on the end; it was not designed for this unusual case where the search term is actually part of a data structure.

 

Here is the relevant code:

 

https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/Controller/CombinedController.php#L246

 

You could perhaps revise this to something like:

 

        case 'External':

            $lookfor = $this->params()->fromQuery('lookfor');

            if (false === strpos($lookfor, '%%lookfor%%')) {

                return $this->redirect()->toUrl($target . urlencode($lookfor));

            } else {

                return $this->redirect()->toUrl(str_replace('%%lookfor%%', urlencode($lookfor), $target));

            }

 

And then in your target[] setting, you would put %%lookfor%% into the URL at the spot where the query string belongs.

 

Does that make sense? If it helps, we can consider making this a standard feature in a future release.

 

- Demian

 

 

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 2:00 PM


To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

Thanks for the detail explanation. Yes you are right that the query string is formatted in an unusual way. So i tried this string where query placed at the end directly in the address bar.

https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "web ontology"}}}}}

And it worked (retrieved 636 articles).

Inspired by this small success I entered this string in the searchbox.ini

target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": ""}}}}}"

But the problem is that VuFind query forwarding mechanism placing the search term at the end and not within the braces.


https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {query_string: {default_operator: AND,query: }}}}}web+ontology

I got stuck here....

Regards

 

On Tue, Mar 21, 2017 at 11:13 PM, Demian Katz <[hidden email]> wrote:

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: Query forwarding (search.ini) for articles@DOAJ

P. S. Mukhopadhyay
Sure Demian. You please take your time. Meanwhile I'm trying and I'll report the outcome.

Regards

On Wed, Mar 22, 2017 at 12:53 AM, Demian Katz <[hidden email]> wrote:

It is possible that there is a logical flaw in my code, since I shared it without actually testing it. I will try to find more time to actually try this out in the next few days; unfortunately this week is quite busy, so I apologize if I do not get to this in as timely a fashion as I would like. If you have time to do some troubleshooting and get it working in the meantime, please share any working version you come up with. Putting some die() statements in strategic places in the code to ensure that it is flowing as expected might be helpful.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 3:21 PM


To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Yes. Changed code and tried all these three possibilities

;target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "%%lookfor%%"}}}}}"
;target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","%%lookfor%%"}}}}}"
target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"%%lookfor%%"}}}}}"

but still query is going after the braces.

https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {%%lookfor%%}}}}}web

Am I doing some mistake in placing %%lookfor%%

 

On Tue, Mar 21, 2017 at 11:44 PM, Demian Katz <[hidden email]> wrote:

Yes, this is the key problem – the VuFind code assumes the search term always goes on the end; it was not designed for this unusual case where the search term is actually part of a data structure.

 

Here is the relevant code:

 

https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/Controller/CombinedController.php#L246

 

You could perhaps revise this to something like:

 

        case 'External':

            $lookfor = $this->params()->fromQuery('lookfor');

            if (false === strpos($lookfor, '%%lookfor%%')) {

                return $this->redirect()->toUrl($target . urlencode($lookfor));

            } else {

                return $this->redirect()->toUrl(str_replace('%%lookfor%%', urlencode($lookfor), $target));

            }

 

And then in your target[] setting, you would put %%lookfor%% into the URL at the spot where the query string belongs.

 

Does that make sense? If it helps, we can consider making this a standard feature in a future release.

 

- Demian

 

 

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 2:00 PM


To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

Thanks for the detail explanation. Yes you are right that the query string is formatted in an unusual way. So i tried this string where query placed at the end directly in the address bar.

https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": "web ontology"}}}}}

And it worked (retrieved 636 articles).

Inspired by this small success I entered this string in the searchbox.ini

target[] = "https://doaj.org/search?source={"query": {"filtered": {"filter": {"bool": {"must": [{"term": {"_type": "article"}}]}}, "query": {"query_string": {"default_operator": "AND","query": ""}}}}}"

But the problem is that VuFind query forwarding mechanism placing the search term at the end and not within the braces.


https://doaj.org/search?source={query: {filtered: {filter: {bool: {must: [{term: {_type: article}}]}}, query: {query_string: {default_operator: AND,query: }}}}}web+ontology

I got stuck here....

Regards

 

On Tue, Mar 21, 2017 at 11:13 PM, Demian Katz <[hidden email]> wrote:

Ahh, yes, I see – when I actually do a search on the site, it does create a very unusual-looking URL with a source parameter set to what looks like a JSON object. I guess this means your original attempt was on the right track; however, I think it was missing the outer braces ({ … }) and was not correctly URL encoded.

 

The problem with trying to use this system with VuFind’s search forwarding is that the current query forwarding mechanism just assumes that the search term needs to be appended to the end of the string. For this situation, the search term actually needs to be embedded at a specific place in the middle of a URL. We would have to add a new feature to support this – for example, some token that can be searched and replaced within the URL string. If you’re interested in trying to implement this, I can find the relevant code and make some more detailed suggestions.

 

In any case, I hope this is helpful – please let me know if I can be of any further assistance.

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Tuesday, March 21, 2017 1:23 PM
To: Demian Katz
Cc: vufind-tech
Subject: Re: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 

Dear Demian

It's not working as desired. I entered

https://doaj.org/search?query_string=medicine&query_field=bibjson.title

And it retrieved all 2,462,260 articles.

 

On Tue, Mar 21, 2017 at 12:45 AM, Demian Katz <[hidden email]> wrote:

Is it possible that this URL is not formatted correctly? It looks like the parameters are being formatted in a Javascript-oriented way, but I wonder if you actually want something that’s using traditional URL parameter formatting, like:

 

?query_string=medicine&query_field=bibjson.title&sort_field=…

 

Does that make a difference?

 

- Demian

 

From: P. S. Mukhopadhyay [mailto:[hidden email]]
Sent: Sunday, March 19, 2017 4:23 PM
To: vufind-tech
Subject: [VuFind-Tech] Query forwarding (search.ini) for articles@DOAJ

 


Dear All

An attempt to forward query to DOAJ for accessing articles against a search term is producing wrong results. In fact it is retrieving all articles (2.4 million+ articles) against the string.

I am using combined interface with the following entry in searchbox.ini:

type[] = External
target[] = "https://doaj.org/search?source=query_string:'medicine',query_field:'bibjson.title',sort_field:'index.unpunctitle.exact',sort_direction:'asc',search_operator:'AND',search_type:'article',page_size:5 "
label[] = Articles@DOAJ

This is based on the syntax as prescribed by DOAJ:
https://doaj.org/widgets

Please help.

--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--

-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------




--
-----------------------------------------------------------------------
Dr. Parthasarathi Mukhopadhyay
Associate Professor, Department of Library and Information Science, University of Kalyani,
Kalyani - 741 235 (WB), India
-----------------------------------------------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech