Many docs updates

Updated known-issues, remove references to 'hash' as appropriate,
clarify script options.

doc/choosing-hash-sizes.txt → doc/choosing-table-sizes.txt

@@ -1,11 +1,11 @@
-=============================
-Choosing hash sizes for khmer
-=============================
+==============================
+Choosing table sizes for khmer
+==============================
 
 If you look at the documentation for the scripts (:doc:`scripts`) you'll
-see two mysterious parameters -- ``-N`` and ``-x``, or, more verbosely,
-``-n_hashes`` and ``--hashsize``.  What are these, and how do you
-specify them?
+see two mysterious parameters -- :option:`-N` and :option:`-x`, or, more
+verbosely, :option:`-n_tables` and :option:`--tablesize`.  What are these, and
+how do you specify them?
 
 The really short version
 ========================
@@ -27,7 +27,7 @@ structure in khmer, which is basically N big hash tables of size x.
 The **product** of the number of hash tables and the size of the hash
 tables specifies the total amount of memory used.
 
-This hash table is used to track k-mers.  If it is too small, khmer
+This table is used to track k-mers.  If it is too small, khmer
 will fail in various ways (and should complain), but there is no harm
 in making it too large. So, **the absolute safest thing to do is to
 specify as much memory as is available**.  Most scripts will inform
@@ -48,8 +48,8 @@ which multiplies out to 128 Gbits of RAM, or 16 Gbytes.
 
 Life is a bit more complicated than this, however, because some scripts --
 load-into-counting and load-graph -- keep ancillary information that will
-consume memory beyond this hash data structure.  So if you run out of
-memory, decrease the hash table size.
+consume memory beyond this table data structure.  So if you run out of
+memory, decrease the table size.
 
 Also see the rules of thumb, below.
 

doc/galaxy.txt

@@ -48,7 +48,7 @@ If not then you may need to `set their datatype manually <https://wiki.galaxypro
 #. After selecting the input files specift if they are paired-interleaved
 or not.
 
-#. Specify the sample type or show the advanced parameters to set the hashsize
-yourself. Consult :doc:`choosing-hash-sizes` for assistance.
+#. Specify the sample type or show the advanced parameters to set the tablesize
+yourself. Consult :doc:`choosing-table-sizes` for assistance.
 
 

doc/guide.txt

@@ -74,10 +74,10 @@ Most scripts *output* fasta, and some mangle headers.  Sorry.  We're
 working on outputting FASTQ for FASTQ input, and removing any header
 mangling.
 
-Picking hash table sizes and k parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Picking k-mer table sizes and k parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For hash table sizes, read :doc:`choosing-hash-sizes`
+For k-mer table sizes, read :doc:`choosing-table-sizes`
 
 For k-mer sizes, we recommend k=20 for digital normalization and k=32
 for partitioning; then assemble with a variety of k parameters.
@@ -197,16 +197,16 @@ you should load in paired end reads, or longer reads, first.
 Iterative and independent normalization
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-You can use --loadhash and --savehash to do iterative normalizations on
-multiple files in multiple steps. For example, break ::
+You can use :option:`--loadtable` and :option:`--savetable` to do iterative
+normalizations on multiple files in multiple steps. For example, break ::
 
   normalize-by-median.py [ ... ] file1.fa file2.fa file3.fa
 
 into multiple steps like so::
 
-  normalize-by-median.py [ ... ] --savehash file1.kh file1.fa
-  normalize-by-median.py [ ... ] --loadhash file1.kh --savehash file2.kh file2.fa
-  normalize-by-median.py [ ... ] --loadhash file2.kh --savehash file3.kh file3.fa
+  normalize-by-median.py [ ... ] --savetable file1.kh file1.fa
+  normalize-by-median.py [ ... ] --loadtable file1.kh --savetable file2.kh file2.fa
+  normalize-by-median.py [ ... ] --loadtable file2.kh --savetable file3.kh file3.fa
 
 The results should be identical!
 

doc/index.txt

@@ -6,9 +6,10 @@
 khmer -- k-mer counting & filtering FTW
 =======================================
 
-:Authors: Michael R. Crusoe, Greg Edvenson, Jordan Fish, Adina Howe, 
-          Eric McDonald, Joshua Nahum, Kaben Nanlohy, Jason Pell, Jared Simpson,
-          Camille Scott, Qingpeng Zhang, and C. Titus Brown
+:Authors: Michael R. Crusoe, Greg Edvenson, Jordan Fish, Adina Howe,
+          Luiz Irber, Eric McDonald, Joshua Nahum, Kaben Nanlohy, Humberto
+          Ortiz-Zuazaga, Jason Pell, Jared Simpson, Camille Scott, Ramakrishnan
+          Rajaram Srinivasan, Qingpeng Zhang, and C. Titus Brown
 
 :Contact: [email protected]
 :License: BSD
@@ -47,14 +48,13 @@ Contents:
    guide
    scripts
    blog-posts
-   choosing-hash-sizes
+   choosing-table-sizes
    partitioning-big-data
    design
    details
    development
    galaxy
    known-issues
-   sequence-access-patterns
    release
    crazy-ideas
    contributors

doc/introduction.txt

@@ -77,7 +77,7 @@ data sets.
 The second most important consideration is memory usage.  The
 effectiveness of all of the Bloom filter-based functions (which is
 everything interesting in khmer!) depends critically on having enough
-memory to do a good job.  See :doc:`choosing-hash-sizes` for more
+memory to do a good job.  See :doc:`choosing-table-sizes` for more
 information.
 
 Copyright and license

doc/known-issues.txt

@@ -14,13 +14,18 @@ https://github.com/ged-lab/khmer/issues/249
 
 If your hashfile gets truncated, perhaps from a full filesystem, then our
 tools currently will get stuck. This is being tracked in 
-https://github.com/ged-lab/khmer/issues/247
+https://github.com/ged-lab/khmer/issues/247 and
+https://github.com/ged-lab/khmer/issues/96 and
+https://github.com/ged-lab/khmer/issues/246
 
 Paired-end reads from Casava 1.8 currently require renaming for use in
 normalize-by-median and abund-filter when used in paired mode. The
 integration of a fix for this is being tracked in
 https://github.com/ged-lab/khmer/issues/23
 
-A user has reported a floating point exception when running
-count-overlap.py. There is no workaround at this time.
-https://github.com/ged-lab/khmer/issues/282
+annotate-partitions.py only outputs FASTA even if given a FASTQ file. This
+issue is being tracked in https://github.com/ged-lab/khmer/issues/46
+
+A user reported that abundance-dist-single.py fails with small files and many
+threads. This issue is being tracked in
+https://github.com/ged-lab/khmer/issues/75

doc/scripts.txt

@@ -7,11 +7,11 @@ khmer's command-line interface
 The simplest way to use khmer's functionality is through the command
 line scripts, located in the scripts/ directory of the khmer
 distribution.  Below is our documentation for these scripts.  Note
-that all scripts can be given '-h' as an option, which will print out
+that all scripts can be given :option:`-h` which will print out
 a list of arguments taken by that script.
 
-Many scripts take '-x' and '-N' parameters, which drive khmer's memory usage.
-These parameters depend on details of your data set; for more information
+Many scripts take :option:`-x` and :option:`-N` parameters, which drive khmer's
+memory usage. These parameters depend on details of your data set; for more information
 on how to choose them, see :doc:`choosing-table-sizes`.
 
 You can also override the default values of :option:`--ksize`/:option:`-k`,

scripts/abundance-dist.py

@@ -30,7 +30,7 @@ def get_parser():
         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 
     parser.add_argument('input_counting_table_filename', help='The name of the'
-                        ' counting table file.')
+                        ' input k-mer counting table file.')
     parser.add_argument('input_sequence_filename', help='The name of the input'
                         ' FAST[AQ] sequence file.')
     parser.add_argument('output_histogram_filename', help='The columns are: '

scripts/annotate-partitions.py

@@ -47,8 +47,11 @@ def get_parser():
 
     parser.add_argument('--ksize', '-k', type=int, default=DEFAULT_K,
                         help="k-mer size (default: %d)" % DEFAULT_K)
-    parser.add_argument('graphbase')
-    parser.add_argument('input_filenames', nargs='+')
+    parser.add_argument('graphbase', help='basename for input and output '
+                        'files')
+    parser.add_argument('input_filenames', metavar='input_sequence_filename',
+                        nargs='+', help='input FAST[AQ] sequences to '
+                        'annotate.')
     parser.add_argument('--version', action='version', version='%(prog)s '
                         + khmer.__version__)
     return parser

scripts/count-median.py

@@ -42,9 +42,12 @@ def get_parser():
         description='Count k-mers summary stats for sequences',
         epilog=textwrap.dedent(epilog))
 
-    parser.add_argument('ctfile', help='input k-mer count table filename')
-    parser.add_argument('input', help='input FAST[AQ] sequence filename')
-    parser.add_argument('output', help='output summary filename')
+    parser.add_argument('ctfile', metavar='input_counting_table_filename',
+                        help='input k-mer count table filename')
+    parser.add_argument('input', metavar='input_sequence_filename',
+                        help='input FAST[AQ] sequence filename')
+    parser.add_argument('output', metavar='output_summary_filename',
+                        help='output summary filename')
     parser.add_argument('--version', action='version', version='%(prog)s '
                         + khmer.__version__)
     return parser

scripts/count-overlap.py

@@ -30,16 +30,19 @@
 
 def get_parser():
     epilog = """
-    An additional report will be written to ${report_filename}.curve containing
-    the increase of overlap k-mers as the number of sequences in the second
-    database increases.
+    An additional report will be written to ${output_report_filename}.curve
+    containing the increase of overlap k-mers as the number of sequences in the
+    second database increases.
     """
     parser = build_hashbits_args(
         descr='Count the overlap k-mers which are the k-mers appearing in two '
         'sequence datasets.', epilog=textwrap.dedent(epilog))
-    parser.add_argument('ptfile', help="input k-mer presence table filename")
-    parser.add_argument('fafile', help="input sequence filename")
-    parser.add_argument('report_filename', help='output report filename')
+    parser.add_argument('ptfile', metavar='input_presence_table_filename',
+                        help="input k-mer presence table filename")
+    parser.add_argument('fafile', metavar='input_sequence_filename',
+                        help="input sequence filename")
+    parser.add_argument('report_filename', metavar='output_report_filename',
+                        help='output report filename')
 
     return parser
 

scripts/do-partition.py

@@ -96,7 +96,8 @@ def get_parser():
                         default=True, action='store_false',
                         help='Keep individual subsets (default: False)')
     parser.add_argument('graphbase', help="base name for output files")
-    parser.add_argument('input_filenames', nargs='+')
+    parser.add_argument('input_filenames', metavar='input_sequence_filename',
+                        nargs='+', help='input FAST[AQ] sequence filenames')
     return parser
 
 

scripts/extract-partitions.py

@@ -59,8 +59,9 @@ def get_parser():
         description="Separate sequences that are annotated with partitions "
         "into grouped files.", epilog=textwrap.dedent(epilog),
         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument('prefix')
-    parser.add_argument('part_filenames', nargs='+')
+    parser.add_argument('prefix', metavar='output_filename_prefix')
+    parser.add_argument('part_filenames', metavar='input_partition_filename',
+                        nargs='+')
     parser.add_argument('--max-size', '-X', dest='max_size',
                         default=DEFAULT_MAX_SIZE, type=int,
                         help='Max group size (n sequences)')

scripts/filter-abund-single.py

@@ -30,12 +30,12 @@
 
 def get_parser():
     epilog = """
-    Trimmed sequences will be placed in ${datafile}.abundfilt.
+    Trimmed sequences will be placed in ${input_sequence_filename}.abundfilt.
 
     This script is constant memory.
 
     To trim reads based on k-mer abundance across multiple files, use
-    :program:`load-into-counting` and :program:`filter-abund`.
+    :program:`load-into-counting.py` and :program:`filter-abund.py`.
 
     Example::
 
@@ -51,7 +51,8 @@ def get_parser():
     parser.add_argument('--savetable', metavar="filename", default='',
                         help="If present, the name of the file to save the "
                         "k-mer counting table to")
-    parser.add_argument('datafile', help="FAST[AQ] sequence file to trim")
+    parser.add_argument('datafile', metavar='input_sequence_filename',
+                        help="FAST[AQ] sequence file to trim")
 
     return parser
 

scripts/filter-abund.py

@@ -28,9 +28,9 @@
 
 def get_parser():
     epilog = """
-    Trimmed sequences will be placed in ${input_filename}.abundfilt for each
-    input sequence file. If the input sequences are from RNAseq or metagenome
-    sequencing then :option:`--variable-coverage` should be used.
+    Trimmed sequences will be placed in ${input_sequence_filename}.abundfilt
+    for each input sequence file. If the input sequences are from RNAseq or
+    metagenome sequencing then :option:`--variable-coverage` should be used.
 
     Example::
 
@@ -40,14 +40,15 @@ def get_parser():
     parser = build_counting_args(
         descr='Trim sequences at a minimum k-mer abundance.',
         epilog=textwrap.dedent(epilog))
-    parser.add_argument('input_table')
-    parser.add_argument('input_filename', nargs='+')
+    parser.add_argument('input_table', metavar='input_presence_table_filename',
+                        help='The input k-mer presence table filename')
+    parser.add_argument('input_filename', metavar='input_sequence_filename',
+                        help='Input FAST[AQ] sequence filename', nargs='+')
     add_threading_args(parser)
     parser.add_argument('--cutoff', '-C', dest='cutoff',
                         default=DEFAULT_CUTOFF, type=int,
                         help="Trim at k-mers below this abundance.")
-
-    parser.add_argument('-V', '--variable-coverage', action='store_true',
+    parser.add_argument('--variable-coverage', '-V', action='store_true',
                         dest='variable_coverage', default=False,
                         help='Only trim low-abundance k-mers from sequences '
                         'that have high coverage.')
@@ -56,7 +57,7 @@ def get_parser():
                         ' k-mer abundance.',
                         default=DEFAULT_NORMALIZE_LIMIT)
     parser.add_argument('-o', '--out', dest='single_output_filename',
-                        default='', metavar="optional output filename",
+                        default='', metavar="optional_output_filename",
                         help='Output the trimmed sequences into a single file '
                         'with the given filename instead of creating a new '
                         'file for each input file.')

scripts/filter-stoptags.py

@@ -37,10 +37,11 @@ def get_parser():
         description="Trim sequences at stoptags.",
         epilog=textwrap.dedent(epilog),
         formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument('-k', default=DEFAULT_K, type=int, help='k-mer size',
-                        dest='ksize')
-    parser.add_argument('stoptags_file')
-    parser.add_argument('input_filenames', nargs='+')
+    parser.add_argument('--ksize', '-k', default=DEFAULT_K, type=int,
+                        help='k-mer size')
+    parser.add_argument('stoptags_file', metavar='input_stoptags_filename')
+    parser.add_argument('input_filenames', metavar='input_sequence_filename',
+                        nargs='+')
     parser.add_argument('--version', action='version', version='%(prog)s '
                         + khmer.__version__)
     return parser

scripts/find-knots.py

@@ -47,9 +47,9 @@ def get_parser():
     Load an k-mer presence table/tagset pair created by load-graph, and a set
     of pmap files created by partition-graph. Go through each pmap file,
     select the largest partition in each, and do the same kind of traversal as
-    in make-initial-stoptags from each of the waypoints in that partition; this
-    should identify all of the HCKs in that partition. These HCKs are output to
-    <graphbase>.stoptags after each pmap file.
+    in :program:`make-initial-stoptags.py` from each of the waypoints in that
+    partition; this should identify all of the HCKs in that partition. These
+    HCKs are output to <graphbase>.stoptags after each pmap file.
 
     Parameter choice is reasonably important. See the pipeline in
     :doc:`partitioning-big-data` for an example run.
@@ -70,7 +70,8 @@ def get_parser():
     parser.add_argument('--min-tablesize', '-x', type=float,
                         default=DEFAULT_COUNTING_HT_SIZE, help='lower bound on'
                         ' the size of the k-mer counting table(s)')
-    parser.add_argument('graphbase')
+    parser.add_argument('graphbase', help='Basename for the input and output '
+                        'files.')
     parser.add_argument('--version', action='version', version='%(prog)s '
                         + khmer.__version__)
     return parser

scripts/load-graph.py

@@ -32,8 +32,11 @@ def get_parser():
     parser.add_argument('--no-build-tagset', '-n', default=False,
                         action='store_true', dest='no_build_tagset',
                         help='Do NOT construct tagset while loading sequences')
-    parser.add_argument('output_filename')
-    parser.add_argument('input_filenames', nargs='+')
+    parser.add_argument('output_filename',
+                        metavar='output_presence_table_filename', help='output'
+                        ' k-mer presence table filename.')
+    parser.add_argument('input_filenames', metavar='input_sequence_filename',
+                        nargs='+', help='input FAST[AQ] sequence filename')
     return parser
 
 

scripts/make-initial-stoptags.py

@@ -59,7 +59,8 @@ def get_parser():
                         help='Set subset size (default 1e4 is prob ok)')
     parser.add_argument('--stoptags', '-S', metavar='filename', default='',
                         help="Use stoptags in this file during partitioning")
-    parser.add_argument('graphbase')
+    parser.add_argument('graphbase', help='basename for input and output '
+                        'filenames')
     return parser
 
 

scripts/merge-partitions.py

@@ -31,14 +31,16 @@ def get_parser():
     Take the ${graphbase}.subset.#.pmap files and merge them all into a single
     ${graphbase}.pmap.merged file for :program:`annotate-partitions.py` to use.
     """
-    parser = argparse.ArgumentParser(description="Merge pmap files.",
-                                     epilog=textwrap.dedent(epilog))
+    parser = argparse.ArgumentParser(
+        description="Merge partition map '.pmap' files.",
+        epilog=textwrap.dedent(epilog))
     parser.add_argument('--ksize', '-k', type=int, default=DEFAULT_K,
                         help="k-mer size (default: %d)" % DEFAULT_K)
     parser.add_argument('--keep-subsets', dest='remove_subsets',
                         default=True, action='store_false',
                         help='Keep individual subsets (default: False)')
-    parser.add_argument('graphbase')
+    parser.add_argument('graphbase', help='basename for input and output '
+                        'files')
     parser.add_argument('--version', action='version', version='%(prog)s '
                         + khmer.__version__)
     return parser