Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gather and manysearch benchmarks for plugin v0.9.6 and v0.9.8 #479

Open
ctb opened this issue Oct 16, 2024 · 8 comments
Open

gather and manysearch benchmarks for plugin v0.9.6 and v0.9.8 #479

ctb opened this issue Oct 16, 2024 · 8 comments

Comments

@ctb
Copy link
Collaborator

ctb commented Oct 16, 2024 •
edited
Loading

gather benchmarks

CURRENT: v0.9.8 (link)

prefix s max_rss
fastmultigather_rocksdb 94.9313 482.69
fastgather 121.758 7676.93
fastmultigather 408.682 22902.7
pygather 1902.84 13743

Cost of RocksDB indexing: 2:40:31 / 9632s / 14.4 GB.

PREVIOUS: v0.9.6 (link)

prefix s max_rss
fastmultigather_rocksdb 102.103 515.24
fastgather 152.312 13071.1
fastmultigather 441.748 13029.6
pygather 2768.48 13755.2

Cost of RocksDB indexing: 4:47:34 / 17255s / 14.0 GB

manysearch benchmarks

plugin version time memory
v0.8.6 814s 18.3 GB
v0.9.0 1332s 27.1 GB
v0.9.1 1361s 22.4 GB
v0.9.5 1534s 21.1 GB
v0.9.6 1752s 27.2 GB
v0.9.8 1520s 19.9 GB
v0.9.8* 686s 42.8 GB

* this run used sig.zip files and manifest CSVs in the benchmarking.

(more description to follow.)
This was referenced Oct 22, 2024
Open
Open
@ctb
Copy link
Collaborator Author

ctb commented Nov 6, 2024 •
edited
Loading

With #498 included:

On a large/complex metagenome, with 45k gather matches, at k=31 with a scaled of 1000 and threshold of 50kb, fmg on rocksdb is kind of ridiculously fast?! This is with the gtdb database in ~ctbrown/scratch3/2024-test-fmg-rocksdb-wort.

  • 22 minutes wall time
  • < 50% CPU (so this is IO bound b/c rocksdb)
  • 1.2 GB of RAM

The question remaining is how well does it parallelize, I think...

time -v output

/usr/bin/time -v sourmash scripts fastmultigather /group/ctbrowngrp/irber/data/wort-data/wort-sra/sigs/ERR4869744.sig gtdb-rs214-k31.rocksdb
...
...fastmultigather is done!
        Command being timed: "sourmash scripts fastmultigather /group/ctbrowngrp/irber/data/wort-data/wort-sra/sigs/ERR4869744.sig gtdb-rs214-k31.rocksdb"
        User time (seconds): 511.18
        System time (seconds): 98.55
        Percent of CPU this job got: 46%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 21:51.37
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1183564
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 59333
        Minor (reclaiming a frame) page faults: 2394647
        Voluntary context switches: 2188167
        Involuntary context switches: 10621
        Swaps: 0
        File system inputs: 25485776
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

@ctb
Copy link
Collaborator Author

ctb commented Nov 6, 2024 •
edited
Loading

scripts and code in ~ctbrown/scratch3/2024-test-host-wort-jean

Tried doing a host contamination analysis on a bunch of the small/simple metagenomes that Jean is analyzing - at k=31, scaled=10_000, -c 4, -t 0, against 8 host genomes (human, cow, dog, cat, chicken, mouse, goat, and pig).

List of metagenomes:

ERR346536
ERR2486211
SRR5790649
ERR2486357
ERR346403
ERR2486297
ERR2241825
ERR2815290
ERR5084157
SRR1631382

Time:

  • 38 seconds
  • < 1 GB of RAM

time -v output for manysearch against many host genomes

% 
...
        Command being timed: "sourmash scripts manysearch all-host.mf.csv small-
list.txt -o small-list.csv -c 4 -t 0"
        User time (seconds): 35.37
        System time (seconds): 4.04
        Percent of CPU this job got: 101%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:38.95
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 729948
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1218534
        Voluntary context switches: 3161
        Involuntary context switches: 668
        Swaps: 0
        File system inputs: 0
        File system outputs: 32
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

manysearch output

maybe some minor contam in ERR2241825

query             p_genome avg_abund   p_metag   metagenome name   
--------          -------- ---------   -------   ---------------   
bosTau9.fa.gz        0.0%     1.0       0.2%     SRR5790649
bosTau9.fa.gz        0.0%     1.7       0.0%     ERR2241825
canFam6.fa.gz        0.0%     1.0       0.2%     SRR5790649
canFam6.fa.gz        0.0%    25.0       0.2%     ERR2241825
felCat9.fa.gz        0.0%     1.0       0.2%     SRR5790649
felCat9.fa.gz        0.0%    20.5       0.2%     ERR2241825
galGal6.fa.gz        0.0%     1.0       0.8%     ERR5084157
galGal6.fa.gz        0.0%     1.0       0.2%     SRR5790649
galGal6.fa.gz        0.0%     3.7       0.1%     ERR2486297
galGal6.fa.gz        0.0%    51.5       0.2%     ERR2241825
hg38.all             0.0%     1.0       2.3%     ERR5084157
hg38.all             0.0%     1.5       0.5%     SRR5790649
hg38.all             0.0%    27.5       0.2%     ERR2241825
mm39.fa.gz           0.0%     1.1       5.3%     SRR5790649
mm39.fa.gz           0.0%     1.1       3.7%     ERR346536
mm39.fa.gz           0.0%     1.1       3.3%     ERR346403
mm39.fa.gz           0.0%    34.7       0.2%     ERR2241825
oviAri4.fa.gz        0.0%     1.0       0.2%     SRR5790649
oviAri4.fa.gz        0.0%    25.3       0.2%     ERR2241825
susScr11.fa.gz       0.0%     1.0       0.2%     SRR5790649
susScr11.fa.gz       1.0%     1.8       0.5%     ERR2241825

@ctb
Copy link
Collaborator Author

ctb commented Nov 6, 2024

Next, host contamination analysis on the 10 biggest/most annoying metagenomes that Jean is analyzing - at k=31, scaled=10_000, -c 4, -t 0, against 8 host genomes (human, cow, dog, cat, chicken, mouse, goat, and pig).

List of metagenomes:

ERR4869744
ERR2241714
ERR2135519
SRR11086695
ERR1726689
ERR2241717
ERR2241633
ERR2592280
SRR5788051
ERR2592342

Time:

  • 4 minutes
  • 2.2 GB of RAM

time -v output

        Command being timed: "sourmash scripts manysearch all-host.mf.csv big-li
st.txt -o big-list.csv -c 4 -t 0"
        User time (seconds): 351.70
        System time (seconds): 39.49
        Percent of CPU this job got: 153%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 4:14.03
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2205472
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 13977379
        Voluntary context switches: 7015
        Involuntary context switches: 11476
        Swaps: 0
        File system inputs: 2147928
        File system outputs: 80
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

manysearch output

looks like some pig contamination in ERR2241633 and ERR2241714 but that's it.

query             p_genome avg_abund   p_metag   metagenome name
--------          -------- ---------   -------   ---------------
bosTau9.fa.gz        0.0%   333.4       1.8%     ERR1726689
bosTau9.fa.gz        0.0%    16.3       0.0%     SRR5788051
bosTau9.fa.gz        0.1%     1.6       0.0%     ERR2241714
bosTau9.fa.gz        0.0%     2.6       0.0%     ERR2135519
bosTau9.fa.gz        0.0%     9.9       0.1%     ERR2592342
bosTau9.fa.gz        0.0%     1.7       0.0%     ERR2241717
bosTau9.fa.gz        0.0%     6.0       0.0%     SRR11086695
bosTau9.fa.gz        0.1%     1.5       0.0%     ERR2241633
bosTau9.fa.gz        0.0%     1.9       0.0%     ERR2592280
bosTau9.fa.gz        0.0%    26.4       0.1%     ERR4869744
canFam6.fa.gz        0.0%   172.8       1.9%     ERR1726689
canFam6.fa.gz        0.0%    20.0       0.1%     SRR5788051
canFam6.fa.gz        0.1%     2.4       0.0%     ERR2241714
canFam6.fa.gz        0.0%    11.7       0.0%     ERR2135519
canFam6.fa.gz        0.1%   105.3       1.6%     ERR2592342
canFam6.fa.gz        0.0%     4.1       0.0%     ERR2241717
canFam6.fa.gz        0.0%     3.7       0.0%     SRR11086695
canFam6.fa.gz        0.1%     3.4       0.0%     ERR2241633
canFam6.fa.gz        0.0%    10.9       0.1%     ERR2592280
canFam6.fa.gz        0.0%    18.4       0.1%     ERR4869744
felCat9.fa.gz        0.0%   129.9       1.9%     ERR1726689
felCat9.fa.gz        0.0%    22.7       0.2%     SRR5788051
felCat9.fa.gz        0.1%     2.4       0.0%     ERR2241714
felCat9.fa.gz        0.0%    10.3       0.0%     ERR2135519
felCat9.fa.gz        0.0%   126.0       1.5%     ERR2592342
felCat9.fa.gz        0.0%     4.5       0.0%     ERR2241717
felCat9.fa.gz        0.0%     4.2       0.0%     SRR11086695
felCat9.fa.gz        0.1%     3.5       0.0%     ERR2241633
felCat9.fa.gz        0.0%    14.5       0.1%     ERR2592280
felCat9.fa.gz        0.0%    13.2       0.1%     ERR4869744
galGal6.fa.gz        0.0%   438.8       1.8%     ERR1726689
galGal6.fa.gz        0.0%    89.5       0.2%     SRR5788051
galGal6.fa.gz        0.0%     4.1       0.0%     ERR2241714
galGal6.fa.gz        0.0%    10.1       0.0%     ERR2135519
galGal6.fa.gz        0.1%   179.2       1.5%     ERR2592342
galGal6.fa.gz        0.0%     9.8       0.0%     ERR2241717
galGal6.fa.gz        0.0%     2.7       0.0%     SRR11086695
galGal6.fa.gz        0.0%     8.7       0.0%     ERR2241633
galGal6.fa.gz        0.0%    22.1       0.1%     ERR2592280
galGal6.fa.gz        0.0%    44.4       0.1%     ERR4869744
hg38.all             0.0%   104.7       1.8%     ERR1726689
hg38.all             0.1%     7.1       0.1%     SRR5788051
hg38.all             0.1%     2.0       0.0%     ERR2241714
hg38.all             0.0%     2.6       0.0%     ERR2135519
hg38.all             0.4%    16.2       1.8%     ERR2592342
hg38.all             0.0%     3.4       0.0%     ERR2241717
hg38.all             0.0%     3.0       0.0%     SRR11086695
hg38.all             0.0%     3.2       0.0%     ERR2241633
hg38.all             0.7%     1.8       0.5%     ERR2592280
hg38.all             0.0%    14.4       0.1%     ERR4869744
mm39.fa.gz           0.1%   156.7       3.2%     ERR1726689
mm39.fa.gz           0.1%    12.5       0.1%     SRR5788051
mm39.fa.gz           0.0%     3.9       0.0%     ERR2241714
mm39.fa.gz           0.0%    11.7       0.0%     ERR2135519
mm39.fa.gz           0.1%    94.1       1.6%     ERR2592342
mm39.fa.gz           0.0%     5.9       0.0%     ERR2241717
mm39.fa.gz           0.0%     5.8       0.0%     SRR11086695
mm39.fa.gz           0.0%     5.9       0.0%     ERR2241633
mm39.fa.gz           0.0%    16.3       0.2%     ERR2592280
mm39.fa.gz           0.0%    10.5       0.1%     ERR4869744
oviAri4.fa.gz        0.0%   298.4       1.8%     ERR1726689
oviAri4.fa.gz        0.0%    39.2       0.1%     SRR5788051
oviAri4.fa.gz        0.1%     2.0       0.0%     ERR2241714
oviAri4.fa.gz        0.0%     5.5       0.0%     ERR2135519
oviAri4.fa.gz        0.0%   214.5       1.5%     ERR2592342
oviAri4.fa.gz        0.0%     2.8       0.0%     ERR2241717
oviAri4.fa.gz        0.0%     2.7       0.0%     SRR11086695
oviAri4.fa.gz        0.1%     2.6       0.0%     ERR2241633
oviAri4.fa.gz        0.0%    19.9       0.1%     ERR2592280
oviAri4.fa.gz        0.0%    24.6       0.1%     ERR4869744
susScr11.fa.gz       0.0%   231.7       1.8%     ERR1726689
susScr11.fa.gz       0.0%    18.5       0.1%     SRR5788051
susScr11.fa.gz       5.8%     1.4       1.1%     ERR2241714
susScr11.fa.gz       0.0%     5.3       0.0%     ERR2135519
susScr11.fa.gz       0.1%    73.1       1.7%     ERR2592342
susScr11.fa.gz       0.7%     1.2       0.2%     ERR2241717
susScr11.fa.gz       0.0%     6.0       0.0%     SRR11086695
susScr11.fa.gz       4.1%     1.3       1.0%     ERR2241633
susScr11.fa.gz       0.0%    10.0       0.1%     ERR2592280
susScr11.fa.gz       0.0%    17.2       0.1%     ERR4869744

@ctb
Copy link
Collaborator Author

ctb commented Nov 7, 2024

Tried out fmg + rocksdb on the same list here ("small-list") -

        Command being timed: "sourmash scripts fastmultigather small-list.txt gtdb-rs214-k31.rocksdb -o small-list-2.csv -c 4 -s 10000"
        User time (seconds): 3519.27
        System time (seconds): 206.97
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:11:36
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1337340
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 20102
        Minor (reclaiming a frame) page faults: 61493492
        Voluntary context switches: 2902992
        Involuntary context switches: 20483
        Swaps: 0
        File system inputs: 37158376
        File system outputs: 6104
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

so, very much NOT parallel 😅 , but about 7 minutes per metagenome.

Next up - can I parallelize with snakemake?

@ctb
Copy link
Collaborator Author

ctb commented Nov 7, 2024 •
edited
Loading

snakemake results. I think the reason for lack of parallelization here is that all but one of the jobs finished ~instantly, with ERR2241825 taking up all the time.

This was done with fmg -c 1 -s 10000.

        Command being timed: "snakemake -c 10"
        User time (seconds): 2674.06
        System time (seconds): 150.74
        Percent of CPU this job got: 92%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 51:08.76
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1445380
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 24
        Minor (reclaiming a frame) page faults: 20091183
        Voluntary context switches: 806820
        Involuntary context switches: 16460
        Swaps: 0
        File system inputs: 10389840
        File system outputs: 6464
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

@ctb
Copy link
Collaborator Author

ctb commented Nov 7, 2024

with -s 1000 and -t 0:

        Command being timed: "snakemake -c 10"
        User time (seconds): 8519.17
        System time (seconds): 1393.78
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:46:46
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1522648
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 3718
        Minor (reclaiming a frame) page faults: 447439486
        Voluntary context switches: 471147
        Involuntary context switches: 109094
        Swaps: 0
        File system inputs: 8305952
        File system outputs: 19944
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

@ctb
Copy link
Collaborator Author

ctb commented Nov 8, 2024

snakemake -c 5 on the 10 biggest/most annoying: good parallelism! Used fastmultigather -c 1 -s 10_000 -t 50_000. Ends up being about 15 minutes per metagenome on average.

Complete log: .snakemake/log/2024-11-07T125121.250529.snakemake.log
        Command being timed: "snakemake -c 5"
        User time (seconds): 29366.88
        System time (seconds): 4355.92
        Percent of CPU this job got: 393%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:22:59
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2137196
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 923
        Minor (reclaiming a frame) page faults: 1116769173
        Voluntary context switches: 1090089
        Involuntary context switches: 354714
        Swaps: 0
        File system inputs: 14801424
        File system outputs: 51544
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

@ctb ctb mentioned this issue Nov 8, 2024
Open
@ctb
Copy link
Collaborator Author

ctb commented Nov 9, 2024

branchwater plugin v0.9.9, indexing GTDB rs220 at k=21 - 7 hours and 20 GB RAM. 25 GB on disk. (located on farm: ~ctbrown/scratch3/2024-test-fmg-rocksdb-wort.)

        Command being timed: "sourmash scripts index -k 21 /group/ctbrowngrp/sourmash-db/gtdb-rs220/gtdb-rs220-k21.zip -o gtdb-rs220-k21.rocksdb"
        User time (seconds): 50407.79
        System time (seconds): 45796.54
        Percent of CPU this job got: 366%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 7:17:41
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 19729860
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 134373
        Minor (reclaiming a frame) page faults: 81083834
        Voluntary context switches: 2051923985
        Involuntary context switches: 2329043
        Swaps: 0
        File system inputs: 276807216
        File system outputs: 485025648
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ctb