Failure in match.log or matchmerge

[batzler]

Description of the bug

I'm running batches of 300 PGSC catalog score files vs an imputed set of biobank target data that I have.
I'm running into issues where some batches of 300 finish without error while others fail. I'm using the same settings and same server/head node setups with the same slurm configurations. When looking at batches of 300 that both finish and fail, they both have SCORE files that fail the matching while others satisfy the matching so its not specific to SCORE files failing the matching. My understanding is that the pipeline handles this just fine. It appears this example of a failed pipeline makes it through the matching for each SCORE file but then fails the matchmerge part.

I'm unable to interpret what the error. Any help is greatly appreciated.

Command used and terminal output

nextflow run pgscatalog/pgsc_calc -profile singularity --input ${samplesheet} --scorefile "${scorefile}" --outdir ${outdir} -r ${pgsc_calc_version} -c ${project}/nxf_config.config --target_build ${target_build} --genotypes_cache $cachedir --run_ancestry $ancestry_file -with-report report.html -with-dag flowchart.png

Terminal error output is quite long but here are the bottom 15 lines or so.
  pgscatalog.match.lib._match.filter: 2024-12-28 10:53:32 DEBUG    Score PGS000929_hmPOS_GRCh38 passes minimum matching threshold (83.33%  variants match)
  pgscatalog.match.lib._match.filter: 2024-12-28 10:53:32 DEBUG    Score PGS000839_hmPOS_GRCh38 passes minimum matching threshold (84.47%  variants match)
  pgscatalog.match.lib._match.log: 2024-12-28 10:53:32 DEBUG    Aggregating best matches into a summary table
  .command.sh: line 9:  1457 Killed                  pgscatalog-matchmerge --dataset pgImputedMerged --scorefile scorefiles.txt.gz --matches *.ipc.zst --min_overlap 0.75 --filter_IDs filter_ids.txt.gz --outdir $PWD --combined -v

Work dir:
  /research/bsi/projects/biobank/s212600.regeneron/batzler/pgsc_calc/NFMERGEDAncestry000700_000999/work/b9/1b0d2d32abf92199a0c47644aea714

Container:
  /research/bsi/projects/biobank/s212600.regeneron/batzler/pgsc_calc/NFMERGEDAncestry000700_000999/cache_v2.0.0-beta/ghcr.io-pgscatalog-pygscatalog-pgscatalog-utils-1.1.2-singularity.img

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
ERROR ~ ERROR: Matching subworkflow failed

 -- Check '.nextflow.log' file for details
ERROR ~ ERROR: No scores calculated!

 -- Check '.nextflow.log' file for details


THEN THERE IS THIS DETAIL IN THE

Relevant files

No response

System information

CentOS Linux
pgsc_calc_version="v2.0.0-beta"
module load nextflow/24.10.3
module load apptainer/1.2.5

[smlmbrt]

Are you starting these all from the same directory and sharing the same work folder? That may cause problems.

Also, what is the error code? To me the killed command is likely to do with memory as it seems like the scoring files are ok.

[batzler]

Thanks for the response. I also believe this to be an memory issue. I tested the same scoring files but in smaller batches and they ran just fine. I will be testing the same larger batch now with increased memory allocations to see if that solves the issue. Thank and no need for any more reply. From: Sam Lambert ***@***.***> Sent: Monday, January 6, 2025 7:46 AM To: PGScatalog/pgsc_calc ***@***.***> Cc: Batzler, Anthony ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] Re: [PGScatalog/pgsc_calc] Failure in match.log or matchmerge (Issue #400) Are you starting these all from the same directory and sharing the same work folder? Also, what is the error code? To me the killed command is likely to do with memory as it seems like the scoring files are ok. - Reply to this email directly, view it on GitHub<#400 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS4PKZV7EAVL4JOUBL7ZRC32JKCKHAVCNFSM6AAAAABUKA3X2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZTGE2DIMRXGM>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[smlmbrt]

Will close the issue then, do reopen if you continue to have problems!