json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[humbleflowers]

I run this command with the new version of magnet
python magnet.py -i ZYMO_D6311_14.nanoq.10.1000.fastq.gz -m ont -o ZYMO_D6311_14.magnet_1 --threads 30 -c relative_abundance.tsv
i get this error

min_abundance: 0
num_species: 62
2678528          GCF_028867355.1         Listeria sp. LM90SB2                    LM90SB2         Complete Genome
2545800          GCF_013343155.1         Pseudomonas sp. FDAARGOS_761            FDAARGOS..      Complete Genome
701198           GCF_016464635.1         Pseudomonas sp. P22                     P22             Scaffold
3041168          GCF_950101725.1         Pseudomonas sp. T2.1D-1.1                               Contig
1739418          GCF_001813485.1         Pseudomonas sp. HMSC067G02              HMSC067G..      Scaffold
2713500          GCF_014229295.1         Listeria farberi                        FSL L7-0..      Contig
2713501          GCF_031843265.2         Listeria swaminathanii                  UTK S2-0..      Complete Genome
2713502          GCF_014322845.1         Listeria immobilis                      FSL L7-1..      Contig
287              GCF_000006765.1         Pseudomonas aeruginosa PAO1             PAO1            Complete Genome
3062686          GCF_030519405.1         Bacillus sp. C28GYM-DRY-1               C28GYM-D..      Contig
2847010          GCF_018919665.1         Escherichia sp. S2_ASV_4                S2_ASV_4        Contig
2952231          GCF_024714025.1         Pseudomonas sp. 17033095                17033095        Contig
2952232          GCF_024714065.1         Pseudomonas sp. 17053182                17053182        Contig
2807721          GCF_016820235.1         Pseudomonas sp. AFW1                    AFW1            Contig
2807722          GCF_016820175.1         Pseudomonas sp. BIS1                    BIS1            Contig
2952234          GCF_024718335.1         Pseudomonas sp. 17053418                17053418        Contig
2952237          GCF_024714085.1         Pseudomonas sp. 17023671                17023671        Contig
2952238          GCF_024714105.1         Pseudomonas sp. 17053703                17053703        Contig
2952239          GCF_024714125.1         Pseudomonas sp. 17104299                17104299        Contig
2952240          GCF_024714155.1         Pseudomonas sp. 17073326                17073326        Contig
562              GCF_000008865.2         Escherichia coli O157:H7 str..          Sakai su..      Complete Genome
1581109          GCF_001808305.1         Pseudomonas sp. HMSC16B01               HMSC16B0..      Scaffold
2952245          GCF_024718435.1         Pseudomonas sp. 18102011                18102011        Contig
2952247          GCF_024718465.1         Pseudomonas sp. 18082551                18082551        Contig
Traceback (most recent call last):
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 353, in <module>
    main()
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 350, in main
    run_magnet(args)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 215, in run_magnet
    reference_metadata = prepare_reference_genomes(valid_taxids, working_directory, ncbi_taxa_db, accession_flag=accession_flag, mag_flag=mag_flag)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 173, in prepare_reference_genomes
    download_result.append(download_reference_genome(taxid, working_dir, mag_flag=mag_flag))
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 65, in download_reference_genome
    res = run_datasets_summary(taxid, ['--reference', '--assembly-source', 'refseq'], mag_flag=mag_flag)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 35, in run_datasets_summary
    return json.loads(grepOut.stdout.strip())
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[humbleflowers]

Hello developer, i got output when i run this command again. It looks like the step downloading genomes breaks and causes the error above.
I got magnet output, i want to calculate abundance. Can you guide me how to estimate the abundance with magnet output file similar to what was done in bioarxiv manuscript?

[beleafs]

Hello developer, i got output when i run this command again. It looks like the step downloading genomes breaks and causes the error above. I got magnet output, i want to calculate abundance. Can you guide me how to estimate the abundance with magnet output file similar to what was done in bioarxiv manuscript?

Could you please sent me the input relative_abundance.tsv so I can check the specific problem? From the error above it's most likely the NCBI database returned a corrupted/empty response the query.

[humbleflowers]

2870764          GCF_019753875.1         Escherichia sp. NT45                    NT45            Contig
2870765          GCF_019753835.1         Escherichia sp. NIC3-2                  NIC3-2          Contig
2870766          GCF_019753755.1         Escherichia sp. NIC4-1                  NIC4-1          Contig
2028568          GCF_002289525.1         Bacillus sp. 7705b                      7705b           Contig
2870767          GCF_019753705.1         Escherichia sp. NIC5-1                  NIC5-1          Contig
1279005          GCF_000408845.1         Escherichia sp. KTE172                  KTE172          Scaffold
1920032          Genome Not Found.
Traceback (most recent call last):
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 353, in <module>
    main()
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 350, in main
    run_magnet(args)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py", line 215, in run_magnet
    reference_metadata = prepare_reference_genomes(valid_taxids, working_directory, ncbi_taxa_db, accession_flag=accession_flag, mag_flag=mag_flag)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 173, in prepare_reference_genomes
    download_result.append(download_reference_genome(taxid, working_dir, mag_flag=mag_flag))
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 80, in download_reference_genome
    res = run_datasets_summary(taxid, [], assembly_level, mag_flag=mag_flag)
  File "/home/vinay.kusuma/pfs-bix-prod/lemur/magnet/utils/reference_finder.py", line 35, in run_datasets_summary
    return json.loads(grepOut.stdout.strip())
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/ubuntu/miniforge3/envs/vk_tools/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[Sat Aug 24 14:44:52 2024]
Error in rule magnet:
    jobid: 36
    input: /home/vinay.kusuma/pfs-bix-prod/ont_zymo_samples/zymo_hmw_r104.fastq.gz, /home/vinay.kusuma/pfs-bix-prod/ont_wwm_pipeline/ONT_ZYMO_WWM_benchmark/zymo_hmw_r104/lemur
    output: /home/vinay.kusuma/pfs-bix-prod/ont_wwm_pipeline/ONT_ZYMO_WWM_benchmark/zymo_hmw_r104/magnet
    shell:
        python /home/vinay.kusuma/pfs-bix-prod/lemur/magnet/magnet.py -i /home/vinay.kusuma/pfs-bix-prod/ont_zymo_samples/zymo_hmw_r104.fastq.gz -m ont -o /home/vinay.kusuma/pfs-bix-prod/ont_wwm_pipeline/ONT_ZYMO_WWM_benchmark/zymo_hmw_r104/magnet --threads 30 -c /home/vinay.kusuma/pfs-bix-prod/ont_wwm_pipeline/ONT_ZYMO_WWM_benchmark/zymo_hmw_r104/lemur/relative_abundance.tsv
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Hello @beleafs Please take a look
relative_abundance.txt

i added a .txt file here as github doesnt accept .tsv

Thank you.

[AlexKoeppelSigSci]

I'm getting the exact same error attempting to run a fresh clone of Magnet

python magnet.py -c filt_test.txt -i barcode01.fastq.gz -o test1 -m ont -t 4 -a 0 --min-abundance 0.001
min_abundance: 0.001
num_species: 1051
Traceback (most recent call last):
  File "/data/home/akoeppel/software/magnet/magnet.py", line 353, in <module>
    main()
  File "/data/home/akoeppel/software/magnet/magnet.py", line 350, in main
    run_magnet(args)
  File "/data/home/akoeppel/software/magnet/magnet.py", line 215, in run_magnet
    reference_metadata = prepare_reference_genomes(valid_taxids, working_directory, ncbi_taxa_db, accession_flag=accession_flag, mag_flag=mag_flag)
  File "/data/home/akoeppel/software/magnet/utils/reference_finder.py", line 173, in prepare_reference_genomes
    download_result.append(download_reference_genome(taxid, working_dir, mag_flag=mag_flag))
  File "/data/home/akoeppel/software/magnet/utils/reference_finder.py", line 65, in download_reference_genome
    res = run_datasets_summary(taxid, ['--reference', '--assembly-source', 'refseq'], mag_flag=mag_flag)
  File "/data/home/akoeppel/software/magnet/utils/reference_finder.py", line 35, in run_datasets_summary
    return json.loads(grepOut.stdout.strip())
  File "/data/home/akoeppel/miniconda3/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/data/home/akoeppel/miniconda3/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/data/home/akoeppel/miniconda3/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Kraken report file: filt_test.txt

[AlexKoeppelSigSci]

Update: I solved the issue at my end by updating my install of the NCBI datasets tool. I was running version 13, updating to 16.37 solved the problem.