Phyla_AMPHORA writing files into installation directory

[samfux84]

Dear Martin Wu,

I recently installed Phyla_AMPHORA on the HPC cluster of ETH Zurich. Then one of our users reported problems when running the script Phylotyping.pl, because it crashed with the following error:

[samapps@euler01 ~]$ module load gcc/4.8.2 gdc raxml/8.2.4 hmmer/3.1 emboss/6.5.7 perl/5.18.4 phyla_amphora
Now run 'perl-init' to initialize your environment for Perl
[samapps@euler01 ~]$ perl-init
[samapps@euler01 ~]$ Phylotyping.pl 

------------- EXCEPTION: Bio::Root::Exception -------------
MSG: Cannot open file '/cluster/apps/gdc/phyla_amphora/Taxonomy/nodes': Permission denied
STACK: Error::throw
STACK: Bio::Root::Root::throw /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/Root/Root.pm:449
STACK: Bio::DB::Taxonomy::flatfile::_build_index /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy/flatfile.pm:370
STACK: Bio::DB::Taxonomy::flatfile::new /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy/flatfile.pm:138
STACK: Bio::DB::Taxonomy::new /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy.pm:119
STACK: /cluster/apps/gdc/phyla_amphora/Scripts/Phylotyping.pl:59
-----------------------------------------------------------
[samapps@euler01 ~]$ 

After investigating the problem, I figured out that the program tries to write files into the directory $Phyla_AMPHORA_home/Taxonomy, which is part of the installation directory of the software. On a HPC cluster, this is a problem. We strictly separate installation directories from the data, that is created during a job. The installation directory is owned by a technical user, and the users running jobs on the cluster do not have write permission in any of the software installation directories.

Is there a way to redirect the files "nodes", "parents", "id2names" and "names2id" to a different directory in user space ?

[wu-lab-uva]

You can change $tax_dir in Phylotyping.pl to a directory where the user has permission to write.

On Mar 30, 2016, at 3:07 AM, samfux84 <[email protected] mailto:[email protected]> wrote:

Dear Martin Wu,

I recently installed Phyla_AMPHORA on the HPC cluster of ETH Zurich. Then one of our users reported problems when running the script Phylotyping.pl, because it crashed with the following error:

[samapps@euler01 ~]$ module load gcc/4.8.2 gdc raxml/8.2.4 hmmer/3.1 emboss/6.5.7 perl/5.18.4 phyla_amphora
Now run 'perl-init' to initialize your environment for Perl
[samapps@euler01 ~]$ perl-init
[samapps@euler01 ~]$ Phylotyping.pl

------------- EXCEPTION: Bio::Root::Exception -------------
MSG: Cannot open file '/cluster/apps/gdc/phyla_amphora/Taxonomy/nodes': Permission denied
STACK: Error::throw
STACK: Bio::Root::Root::throw /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/Root/Root.pm:449
STACK: Bio::DB::Taxonomy::flatfile::_build_index /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy/flatfile.pm:370
STACK: Bio::DB::Taxonomy::flatfile::new /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy/flatfile.pm:138
STACK: Bio::DB::Taxonomy::new /cluster/apps/gdc/perl5/perls/perl-5.18.4/lib/site_perl/5.18.4/Bio/DB/Taxonomy.pm:119

STACK: /cluster/apps/gdc/phyla_amphora/Scripts/Phylotyping.pl:59

[samapps@euler01 ~]$
After investigating the problem, I figured out that the program tries to write files into the directory $Phyla_AMPHORA_home/Taxonomy, which is part of the installation directory of the software. On a HPC cluster, this is a problem. We strictly separate installation directories from the data, that is created during a job. The installation directory is owned by a technical user, and the users running jobs on the cluster do not have write permission in any of the software installation directories.

Is there a way to redirect the files "nodes", "parents", "id2names" and "names2id" to a different directory in user space ?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub #4

[samfux84]

Thank you very much for the fast reply. I will change $tax_dir.

[samfux84]

I had another look at this issue and figured out that the files are not directly created by Phyla_AMPHORA, but rather by calling the perl function Bio::DB::Taxonomy->new(). Are these files only useful during a single Phyla_APMPHORA run or do users need to keep them for a longer time ?

If they are only useful for the time span of a single job, then I would rather change the Phylotyping.pl script and replace the line

my $taxdb = Bio::DB::Taxonomy->new(     -source   => 'flatfile',
                                        -directory=>$tax_dir,
                                        -nodesfile => "$tax_dir/nodes.dmp",
                                        -namesfile => "$tax_dir/names.dmp");

with

my $taxdb = Bio::DB::Taxonomy->new(     -source   => 'flatfile',
                                        -directory=> $ENV{'TMPDIR'},
                                        -nodesfile => "$tax_dir/nodes.dmp",
                                        -namesfile => "$tax_dir/names.dmp");

$TMPDIR is the temporary directory created by the cluster's batch system (IBM LSF). Can you please comment on this ?

Best regards

[wu-lab-uva]

These files can be used by multiple users and only need to be created once. You should keep them for a longer time.

On Apr 4, 2016, at 4:34 AM, samfux84 [email protected] wrote:

I had another look at this issue and figured out that the files are not directly created by Phyla_AMPHORA, but rather by calling the perl function Bio::DB::Taxonomy->new(). Are these files only useful during a single Phyla_APMPHORA run or do users need to keep them for a longer time ?

If they are only useful for the time span of a single job, then I would rather change the Phylotyping.pl script and replace the line

my $taxdb = Bio::DB::Taxonomy->new( -source => 'flatfile',
-directory=>$tax_dir,
-nodesfile => "$tax_dir/nodes.dmp",
-namesfile => "$tax_dir/names.dmp");
with

my $taxdb = Bio::DB::Taxonomy->new( -source => 'flatfile',
-directory=> $ENV{'TMPDIR'},
-nodesfile => "$tax_dir/nodes.dmp",
-namesfile => "$tax_dir/names.dmp");
$TMPDIR is the temporary directory created by the cluster's batch system (IBM LSF). Can you please comment on this ?

Best regards

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #4 (comment)