Another case of empty distance matrix

[bernt-matthias]

I'm new to drep and also run into a case of empty distance matrix. Checked other issues here but could not find a solution so far.

dRep compare outdir  -g 'consensus02.fna.fasta'    --MASH_sketch '1000' --P_ani 0.9  --primary_chunksize 5000   --S_algorithm 'ANImf'  --n_PRESET 'normal'   --coverage_method 'larger'  --S_ani 0.99 --cov_thresh 0.1  --clusterAlg 'average'    --warn_dist 0.25 --warn_sim 0.98 --warn_aln 0.25 --debug

***************************************************
    ..:: dRep compare Step 1. Cluster ::..
***************************************************
    
Loading genomes from a list
Nevermind! Ill try loading as a genome now
Running primary clustering
Running pair-wise MASH clustering
Traceback (most recent call last):
  File "/usr/local/bin/dRep", line 32, in <module>
    Controller().parseArguments(args)
  File "/usr/local/lib/python3.10/site-packages/drep/controller.py", line 102, in parseArguments
    self.compare_operation(**vars(args))
  File "/usr/local/lib/python3.10/site-packages/drep/controller.py", line 53, in compare_operation
    drep.d_workflows.compare_wrapper(kwargs['work_directory'],**kwargs)
  File "/usr/local/lib/python3.10/site-packages/drep/d_workflows.py", line 97, in compare_wrapper
    drep.d_cluster.controller.d_cluster_wrapper(wd, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 184, in d_cluster_wrapper
    GenomeClusterController(workDirectory, **kwargs).main()
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 32, in main
    self.run_primary_clustering()
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 100, in run_primary_clustering
    Mdb, Cdb, cluster_ret = drep.d_cluster.compare_utils.all_vs_all_MASH(self.Bdb, self.wd.get_dir('MASH'), **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/compare_utils.py", line 115, in all_vs_all_MASH
    Cdb, cluster_ret = cluster_mash_database(Mdb, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/compare_utils.py", line 280, in cluster_mash_database
    Cdb, linkage = drep.d_cluster.cluster_utils.cluster_hierarchical(linkage_db, linkage_method= P_Lmethod, \
  File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/cluster_utils.py", line 114, in cluster_hierarchical
    linkage = scipy.cluster.hierarchy.linkage(arr, method= linkage_method)
  File "/usr/local/lib/python3.10/site-packages/scipy/cluster/hierarchy.py", line 1033, in linkage
    n = int(distance.num_obs_y(y))
  File "/usr/local/lib/python3.10/site-packages/scipy/spatial/distance.py", line 2657, in num_obs_y
    raise ValueError("The number of observations cannot be determined on "
ValueError: The number of observations cannot be determined on an empty distance matrix.

dRep check_dependencies

mash.................................... all good        (location = /usr/local/bin/mash)
nucmer.................................. all good        (location = /usr/local/bin/nucmer)
checkm.................................. all good        (location = /usr/local/bin/checkm)
ANIcalculator........................... !!! ERROR !!!   (location = None)
prodigal................................ all good        (location = /usr/local/bin/prodigal)
centrifuge.............................. !!! ERROR !!!   (location = None)
nsimscan................................ !!! ERROR !!!   (location = None)
fastANI................................. all good        (location = /usr/local/bin/fastANI)
skani................................... all good        (location = /usr/local/bin/skani)

The created log file apparently contains no additional info:

12-20 11:06 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
12-20 11:06 DEBUG    ***Logger started up at /work/songalax/galaxy/database/jobs_directory/000/299/299059/working/outdir/log/logger.log***
12-20 11:06 DEBUG    Command to run dRep was: /usr/local/bin/dRep compare outdir -g consensus02.fna.fasta --MASH_sketch 1000 --P_ani 0.9 --primary_chunksize 5000 --S_algorithm ANImf --n_PRESET normal --coverage_method larger --S_ani 0.99 --cov_thresh 0.1 --clusterAlg average --warn_dist 0.25 --warn_sim 0.98 --warn_aln 0.25 --debug

12-20 11:06 DEBUG    dRep version 3.5.0 was run 

12-20 11:06 DEBUG    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

12-20 11:06 DEBUG    Namespace(operation='compare', work_directory='outdir', processors=6, debug=True, genomes=['consensus02.fna.fasta'], S_algorithm='ANImf', MASH_sketch='1000', SkipMash=False, SkipSecondary=False, skani_extra='', n_PRESET='normal', P_ani=0.9, S_ani=0.99, cov_thresh=0.1, coverage_method='larger', clusterAlg='average', multiround_primary_clustering=False, primary_chunksize=5000, greedy_secondary_clustering=False, run_tertiary_clustering=False, gen_warnings=False, warn_dist='0.25', warn_sim='0.98', warn_aln='0.25')
12-20 11:06 DEBUG    Starting the compare operation
12-20 11:06 INFO     ***************************************************
    ..:: dRep compare Step 1. Cluster ::..
***************************************************
    
12-20 11:06 INFO     Loading genomes from a list
12-20 11:06 INFO     Nevermind! Ill try loading as a genome now
12-20 11:06 INFO     Running primary clustering
12-20 11:06 INFO     Running pair-wise MASH clustering
12-20 11:06 DEBUG    Clustering MASH database

running drep via singularity using the biuocontainer with tag drep:3.5.0--pyhdfd78af_0

[MrOlm]

The issue is that dRep only thinks you included 1 genome. Each genome to be compared needs to be given as a separate file.Best,MattOn Dec 20, 2024, at 11:09 AM, M Bernt ***@***.***> wrote: I'm new to drep and also run into a case of empty distance matrix. Checked other issues here but could not find a solution so far. dRep compare outdir -g 'consensus02.fna.fasta' --MASH_sketch '1000' --P_ani 0.9 --primary_chunksize 5000 --S_algorithm 'ANImf' --n_PRESET 'normal' --coverage_method 'larger' --S_ani 0.99 --cov_thresh 0.1 --clusterAlg 'average' --warn_dist 0.25 --warn_sim 0.98 --warn_aln 0.25 --debug *************************************************** ..:: dRep compare Step 1. Cluster ::.. *************************************************** Loading genomes from a list Nevermind! Ill try loading as a genome now Running primary clustering Running pair-wise MASH clustering Traceback (most recent call last): File "/usr/local/bin/dRep", line 32, in <module> Controller().parseArguments(args) File "/usr/local/lib/python3.10/site-packages/drep/controller.py", line 102, in parseArguments self.compare_operation(**vars(args)) File "/usr/local/lib/python3.10/site-packages/drep/controller.py", line 53, in compare_operation drep.d_workflows.compare_wrapper(kwargs['work_directory'],**kwargs) File "/usr/local/lib/python3.10/site-packages/drep/d_workflows.py", line 97, in compare_wrapper drep.d_cluster.controller.d_cluster_wrapper(wd, **kwargs) File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 184, in d_cluster_wrapper GenomeClusterController(workDirectory, **kwargs).main() File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 32, in main self.run_primary_clustering() File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/controller.py", line 100, in run_primary_clustering Mdb, Cdb, cluster_ret = drep.d_cluster.compare_utils.all_vs_all_MASH(self.Bdb, self.wd.get_dir('MASH'), **self.kwargs) File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/compare_utils.py", line 115, in all_vs_all_MASH Cdb, cluster_ret = cluster_mash_database(Mdb, **kwargs) File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/compare_utils.py", line 280, in cluster_mash_database Cdb, linkage = drep.d_cluster.cluster_utils.cluster_hierarchical(linkage_db, linkage_method= P_Lmethod, \ File "/usr/local/lib/python3.10/site-packages/drep/d_cluster/cluster_utils.py", line 114, in cluster_hierarchical linkage = scipy.cluster.hierarchy.linkage(arr, method= linkage_method) File "/usr/local/lib/python3.10/site-packages/scipy/cluster/hierarchy.py", line 1033, in linkage n = int(distance.num_obs_y(y)) File "/usr/local/lib/python3.10/site-packages/scipy/spatial/distance.py", line 2657, in num_obs_y raise ValueError("The number of observations cannot be determined on " ValueError: The number of observations cannot be determined on an empty distance matrix. dRep check_dependencies mash.................................... all good (location = /usr/local/bin/mash) nucmer.................................. all good (location = /usr/local/bin/nucmer) checkm.................................. all good (location = /usr/local/bin/checkm) ANIcalculator........................... !!! ERROR !!! (location = None) prodigal................................ all good (location = /usr/local/bin/prodigal) centrifuge.............................. !!! ERROR !!! (location = None) nsimscan................................ !!! ERROR !!! (location = None) fastANI................................. all good (location = /usr/local/bin/fastANI) skani................................... all good (location = /usr/local/bin/skani) The created log file apparently contains no additional info: 12-20 11:06 DEBUG !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 12-20 11:06 DEBUG ***Logger started up at /work/songalax/galaxy/database/jobs_directory/000/299/299059/working/outdir/log/logger.log*** 12-20 11:06 DEBUG Command to run dRep was: /usr/local/bin/dRep compare outdir -g consensus02.fna.fasta --MASH_sketch 1000 --P_ani 0.9 --primary_chunksize 5000 --S_algorithm ANImf --n_PRESET normal --coverage_method larger --S_ani 0.99 --cov_thresh 0.1 --clusterAlg average --warn_dist 0.25 --warn_sim 0.98 --warn_aln 0.25 --debug 12-20 11:06 DEBUG dRep version 3.5.0 was run 12-20 11:06 DEBUG !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 12-20 11:06 DEBUG Namespace(operation='compare', work_directory='outdir', processors=6, debug=True, genomes=['consensus02.fna.fasta'], S_algorithm='ANImf', MASH_sketch='1000', SkipMash=False, SkipSecondary=False, skani_extra='', n_PRESET='normal', P_ani=0.9, S_ani=0.99, cov_thresh=0.1, coverage_method='larger', clusterAlg='average', multiround_primary_clustering=False, primary_chunksize=5000, greedy_secondary_clustering=False, run_tertiary_clustering=False, gen_warnings=False, warn_dist='0.25', warn_sim='0.98', warn_aln='0.25') 12-20 11:06 DEBUG Starting the compare operation 12-20 11:06 INFO *************************************************** ..:: dRep compare Step 1. Cluster ::.. *************************************************** 12-20 11:06 INFO Loading genomes from a list 12-20 11:06 INFO Nevermind! Ill try loading as a genome now 12-20 11:06 INFO Running primary clustering 12-20 11:06 INFO Running pair-wise MASH clustering 12-20 11:06 DEBUG Clustering MASH database running drep via singularity using the biuocontainer with tag drep:3.5.0--pyhdfd78af_0 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>