index-alphabetic-browse and temp dir for sorting

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

index-alphabetic-browse and temp dir for sorting

Tod Olson
vufind-tech,

We're porting to the new 3.1 scripts, and notice that in index-alphabetic-browse.sh the tempdir for sort is hard-coded:

        sort -T /var/tmp ....

In our environments, /var/tmp often is only a few GB, but our largest browse index is 8.1GB. We can certainly hack our custom version of the script to override that, but I'm wondering if it makes sense to do this a little more flexibly, something like:

SORT_TMPDIR=${SORT_TMPDIR:-$SOLR_HOME}
export SORT_TMPDIR

# rest of script

function build_browse
{
    # ...
    sort -T ${SORT_TMPDIR} ...
    # ...
}

Or maybe default to ${SOLR_HOME}/tmp. I would think the SOLR_HOME partition would have plenty of sort space on most installations.

If that hard-coded temp dir isn't a problem for others, I won't worry about it. But if there is a need to make it more flexible, maybe we can find something that works for everyone.

-Tod


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech
Reply | Threaded
Open this post in threaded view
|

Re: index-alphabetic-browse and temp dir for sorting

Demian Katz
Tod,

It appears that the /var/tmp path has been hard-coded since the script was created, so this is not a new problem. I would guess that this was a preference of the National Library of Australia (where the script was first created) which simply made its way into the project.

I'm always in favor of making things configurable when possible, so replacing the hard-coded path with a variable and using a reasonable default makes a lot of sense to me.

I'm not sure how I feel about $SOLR_HOME/tmp as the default location, though -- your argument that this partition is likely to have plenty of space makes sense, but since this is not a directory that normally exists, I'm wondering if we might run into problems with either needing to manually create it in the script, or else with the script leaving behind a weird tmp directory in an unexpected location.

Perhaps a conservative approach would be to default to /var/tmp for backward compatibility, and recommend creating $SOLR_HOME/tmp as a best practice for large indexes.

What do you think?

- Demian

-----Original Message-----
From: Tod Olson [mailto:[hidden email]]
Sent: Tuesday, January 10, 2017 3:52 PM
To: vufind-tech
Subject: [VuFind-Tech] index-alphabetic-browse and temp dir for sorting

vufind-tech,

We're porting to the new 3.1 scripts, and notice that in index-alphabetic-browse.sh the tempdir for sort is hard-coded:

        sort -T /var/tmp ....

In our environments, /var/tmp often is only a few GB, but our largest browse index is 8.1GB. We can certainly hack our custom version of the script to override that, but I'm wondering if it makes sense to do this a little more flexibly, something like:

SORT_TMPDIR=${SORT_TMPDIR:-$SOLR_HOME}
export SORT_TMPDIR

# rest of script

function build_browse
{
    # ...
    sort -T ${SORT_TMPDIR} ...
    # ...
}

Or maybe default to ${SOLR_HOME}/tmp. I would think the SOLR_HOME partition would have plenty of sort space on most installations.

If that hard-coded temp dir isn't a problem for others, I won't worry about it. But if there is a need to make it more flexible, maybe we can find something that works for everyone.

-Tod


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fxeonphi&data=01%7C01%7Cdemian.katz%40villanova.edu%7C2d047113806146cf7c0208d4399d6b13%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=2TLyxk9Fmnjcg32AeHwsnXUHh12qin%2Bv3l0cmL%2F2Qro%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=01%7C01%7Cdemian.katz%40villanova.edu%7C2d047113806146cf7c0208d4399d6b13%7C765a8de5cf9444f09cafae5bf8cfa366%7C1&sdata=zoNvRR44395zfaBQ1hL02OzuFwY8ytdIM0zVMkAvXG4%3D&reserved=0

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Vufind-tech mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/vufind-tech