Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a PoC disk_estimator feature to the vcf_to_bq preprocessor. #335

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add a PoC disk_estimator feature to the vcf_to_bq preprocessor. #335

wants to merge 5 commits into from

Commits on May 13, 2019

  1. Creates PoC pipeline to estimate disk usage of vcf_to_bq on Dataflow. 

    The pipeline uses raw file size and raw+encoded sizes of a short snippet at beginning of VCF files to
estimate the encoded size for a commit. The major blocking bug is that when the snippets are being read from VCFs in an
encoded format, lines are being read more than once.
    @hanjohn
    hanjohn committed May 13, 2019
    Configuration menu
    Copy the full SHA
    8c6638f View commit details
    Browse the repository at this point in the history

  2. Clean up disk_estimator code and style per PR comments.

    @hanjohn
    John Han authored and hanjohn committed May 13, 2019
    Configuration menu
    Copy the full SHA
    4ccd72d View commit details
    Browse the repository at this point in the history

  3. Make final fixes to PR

    @hanjohn
    hanjohn committed May 13, 2019
    Configuration menu
    Copy the full SHA
    68c8dad View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2019

  1. Merge branch 'master' into disk_estimator

    @hanjohn
    hanjohn committed Sep 3, 2019
    Configuration menu
    Copy the full SHA
    ba9ffeb View commit details
    Browse the repository at this point in the history

  2. Make fixes per tneymanov@ comments

    @hanjohn
    hanjohn committed Sep 3, 2019
    Configuration menu
    Copy the full SHA
    18c9807 View commit details
    Browse the repository at this point in the history