Code

feat!: genome feature support#3

Code

Merged

davidlougheed merged 115 commits intomainbento-platform/bento_reference_service:mainfrom feat/genome-featuresbento-platform/bento_reference_service:feat/genome-featuresCopy head branch name to clipboardMay 22, 2024

+4,854-840

Commits

Commits on May 7, 2024

• feat: add gff3 gz uri fields to db + models

  davidlougheed

  committed
  e62f976

  Copy full SHA for e62f976

• chore: add schema, models, db functions for genome features

  davidlougheed

  committed
  7424442

  Copy full SHA for 7424442

• feat(routes): define gff3 / feature API endpoints

  davidlougheed

  committed
  abfe955

  Copy full SHA for abfe955

• chore: work on gff3 parsing code for ingestion

  davidlougheed

  committed
  2c1e01f

  Copy full SHA for 2c1e01f

• chore: stub out db calls for genome feature ingest

  davidlougheed

  committed
  2e1457f

  Copy full SHA for 2e1457f

• fix(schema): missing semicolon

  davidlougheed

  committed
  7a2a7f3

  Copy full SHA for 7a2a7f3

• fix: issues with db interface

  davidlougheed

  committed
  687dfc7

  Copy full SHA for 687dfc7

• lint

  davidlougheed

  committed
  3a136bb

  Copy full SHA for 3a136bb

• chore: work on workflow definitions

  davidlougheed

  committed
  87695c5

  Copy full SHA for 87695c5

• Merge remote-tracking branch 'origin/main' into feat/genome-features

  davidlougheed

  committed
  c7ea14a

  Copy full SHA for c7ea14a

• chore: bump version to 0.2.0

  davidlougheed

  committed
  388b439

  Copy full SHA for 388b439

Commits on May 8, 2024

• feat: allow putting GFF3 files to ingest annotations

  davidlougheed

  committed
  d971fc9

  Copy full SHA for d971fc9

• feat(workflows): optional gff3 ingestion in fasta ingest workflow

  davidlougheed

  committed
  b30747d

  Copy full SHA for b30747d

• fix: workflow issues

  davidlougheed

  committed
  813008c

  Copy full SHA for 813008c

• chore(workflows): implement gff3_annot workflow

  davidlougheed

  committed
  89f2dcb

  Copy full SHA for 89f2dcb

• chore(db): implement bulk feature ingestion

  davidlougheed

  committed
  5a7c310

  Copy full SHA for 5a7c310

• chore: add default tmp folder

  davidlougheed

  committed
  3b0b9bd

  Copy full SHA for 3b0b9bd

• chore: create ingest tmp dir if needed

  davidlougheed

  committed
  94efe08

  Copy full SHA for 94efe08

• fix: mismatch between DB and Pydantic model for strand

  davidlougheed

  committed
  ccb5563

  Copy full SHA for ccb5563

• chore: properly ignore contents of tmp folder

  davidlougheed

  committed
  60a5515

  Copy full SHA for 60a5515

• fix: misc issues with GFF feature ingestion

  davidlougheed

  committed
  44ef1fa

  Copy full SHA for 44ef1fa

• fix: missing enumerate() in feature ingest

  davidlougheed

  committed
  3aabd0c

  Copy full SHA for 3aabd0c

• fix: clear genome features before (re)ingesting them

  davidlougheed

  committed
  77991c6

  Copy full SHA for 77991c6

• fix: issues with ingesting features to db + ingest feature types

  davidlougheed

  committed
  57be8f5

  Copy full SHA for 57be8f5

• chore(sql): don't create unneeded table, add v0.2 migration

  davidlougheed

  committed
  8229bd9

  Copy full SHA for 8229bd9

• chore: don't log everything as debug

  davidlougheed

  committed
  e764d76

  Copy full SHA for e764d76

• fix: issues with feature ingest

  davidlougheed

  committed
  a63dcfe

  Copy full SHA for a63dcfe

• test: genome feature testing

  davidlougheed

  committed
  203ca58

  Copy full SHA for 203ca58

• fix(workflows): issues with fasta_ref workflow

  Show description for e99b2bf
  davidlougheed

  committed
  e99b2bf

  Copy full SHA for e99b2bf

• fix(workflows): GFF3 pattern for fasta_ref workflow

  davidlougheed

  committed
  d9842bf

  Copy full SHA for d9842bf

• fix: undo debug thing

  davidlougheed

  committed
  820b603

  Copy full SHA for 820b603

Commits on May 9, 2024

• fix(features): use contig-batches to keep parents with child features

  davidlougheed

  committed
  2296e30

  Copy full SHA for 2296e30

• chore: more logging for feature ingest

  davidlougheed

  committed
  1e76c7b

  Copy full SHA for 1e76c7b

• fix(db): syntax for feature_id in-clause

  davidlougheed

  committed
  b9344e3

  Copy full SHA for b9344e3

• fix: logger config

  davidlougheed

  committed
  dd4ef44

  Copy full SHA for dd4ef44

• chore: log time taken to ingest batch of feaures

  davidlougheed

  committed
  6572837

  Copy full SHA for 6572837

• chore(db): implement attribute fetch, fix offset/limit perf issue

  davidlougheed

  committed
  c5bb409

  Copy full SHA for c5bb409

Commits on May 10, 2024

• perf: add some missing SQL indices

  davidlougheed

  committed
  2ae4ee3

  Copy full SHA for 2ae4ee3

• fix: handle missing genome when ingesting annotations

  davidlougheed

  committed
  cc00733

  Copy full SHA for cc00733

• refact!: rewrite schema for better feature storage/indexing

  davidlougheed

  committed
  d3af68e

  Copy full SHA for d3af68e

• fix: issue with v0.2 migration SQL

  davidlougheed

  committed
  c432588

  Copy full SHA for c432588

• lint

  davidlougheed

  committed
  edf6e9c

  Copy full SHA for edf6e9c

• perf(db): release connections earlier in db manager

  davidlougheed

  committed
  938f6b5

  Copy full SHA for 938f6b5

• fix(workflows): fix jumping the gun ingesting fasta+gff3 at once

  davidlougheed

  committed
  e8e5bed

  Copy full SHA for e8e5bed

• refact!: rewrite feature ingest to use async task-based flow

  Show description for 6d9ad33
  davidlougheed

  committed
  6d9ad33

  Copy full SHA for 6d9ad33

• chore: increase log interval for feature ingest

  davidlougheed

  committed
  dd6b998

  Copy full SHA for dd6b998

• refact: cleanup genomes router exc fn used once

  davidlougheed

  committed
  c5d6165

  Copy full SHA for c5d6165

Commits on May 11, 2024

• chore: update gff3 ingest workflow for new task system

  davidlougheed

  committed
  b416d5c

  Copy full SHA for b416d5c

• fix: bad task status literal in Task model

  davidlougheed

  committed
  122c40d

  Copy full SHA for 122c40d

• fix(db): issues with genome feature select query

  davidlougheed

  committed
  1c9c34a

  Copy full SHA for 1c9c34a

Commits on May 13, 2024

• perf: add missing indices to aid in querying+deletion

  davidlougheed

  committed
  ea3009f

  Copy full SHA for ea3009f

• feat: allow deleting features via tested endpoint

  davidlougheed

  committed
  75c4983

  Copy full SHA for 75c4983

• fix(schema): missing on delete cascade for tasks genome fk

  davidlougheed

  committed
  ee19841

  Copy full SHA for ee19841

• chore: update lockfile

  davidlougheed

  committed
  140810a

  Copy full SHA for 140810a

Commits on May 14, 2024

• fix: genome alias handling

  davidlougheed

  committed
  2089a4b

  Copy full SHA for 2089a4b

• fix(db): contig alias deserialization

  davidlougheed

  committed
  a4dc701

  Copy full SHA for a4dc701

• chore(db): update queued/running tasks to error on db startup

  davidlougheed

  committed
  dac9a7c

  Copy full SHA for dac9a7c

• chore(workflows): update fasta workflow gff3 ingest fn

  davidlougheed

  committed
  cdc781c

  Copy full SHA for cdc781c

• chore(workflows): don't output any artifacts

  davidlougheed

  committed
  16298c6

  Copy full SHA for 16298c6

• chore(deps): add fasta-checksum-utils as a dev util

  davidlougheed

  committed
  35a855e

  Copy full SHA for 35a855e

• test: add hg38 chr1:1-100000 test "genome"

  davidlougheed

  committed
  e06649c

  Copy full SHA for e06649c

• fix(db): alias deserialization

  davidlougheed

  committed
  380775a

  Copy full SHA for 380775a

• test: fix async db tests

  davidlougheed

  committed
  2d50000

  Copy full SHA for 2d50000

• test: creation of covid + hg38 subset genomes

  davidlougheed

  committed
  f47c761

  Copy full SHA for f47c761

• test: iter test for features for hg38 subset + sars cov 2

  davidlougheed

  committed
  49c3a80

  Copy full SHA for 49c3a80

• fix(features): not properly removing Parent attr on ingest

  davidlougheed

  committed
  e5037ef

  Copy full SHA for e5037ef

• chore(features): move standard GFF Name attr to constant

  davidlougheed

  committed
  8339def

  Copy full SHA for 8339def

• test: fix issues with test gencode file

  davidlougheed

  committed
  0d4097a

  Copy full SHA for 0d4097a

• chore(db): log feature types being ingested

  davidlougheed

  committed
  bd64c4f

  Copy full SHA for bd64c4f

• test: fix ssl errors in streaming response tests

  davidlougheed

  committed
  bf80fc3

  Copy full SHA for bf80fc3

• chore(db): log better error message when feature has missing parent

  davidlougheed

  committed
  2cc6044

  Copy full SHA for 2cc6044

• test: start testing querying of genome features

  davidlougheed

  committed
  07b8249

  Copy full SHA for 07b8249

• docs: update status of annotation service

  davidlougheed

  committed
  e0611ef

  Copy full SHA for e0611ef

• fix(streaming): deprecation notice for streaming tcp connector init

  davidlougheed

  committed
  3a7cedd

  Copy full SHA for 3a7cedd

• chore(db): consistent use of jsonb

  davidlougheed

  committed
  3ca266e

  Copy full SHA for 3ca266e

• fix(db): correctly deduplicate attribute keys/value lookups

  davidlougheed

  committed
  ce0317c

  Copy full SHA for ce0317c

• chore(db): create GIN index on genome feature attr values

  davidlougheed

  committed
  e2122e6

  Copy full SHA for e2122e6

Commits on May 15, 2024

• test: use mock instead of real site for streaming tests

  davidlougheed

  committed
  1932d28

  Copy full SHA for 1932d28

• test: feature_types summary route

  davidlougheed

  committed
  bb93de5

  Copy full SHA for bb93de5

• chore(features): don't keep Name attribute in feature attrs

  davidlougheed

  committed
  6032870

  Copy full SHA for 6032870

• fix(db): issues with fetching contigs by checksum

  davidlougheed

  committed
  5583bc8

  Copy full SHA for 5583bc8

• refact: separate query and filter genome feature fns

  davidlougheed

  committed
  3190f30

  Copy full SHA for 3190f30

• test: more db tests

  davidlougheed

  committed
  bf2999a

  Copy full SHA for bf2999a

• fix: genome feature endpoints - either query or filter for now

  Show description for ba52f99
  davidlougheed

  committed
  ba52f99

  Copy full SHA for ba52f99

• fix(features): fix bad feature name extraction in some cases

  davidlougheed

  committed
  b2dd332

  Copy full SHA for b2dd332

• test: filter genome features based on name

  davidlougheed

  committed
  b2bf066

  Copy full SHA for b2bf066

• lint

  davidlougheed

  committed
  2ca7bbb

  Copy full SHA for 2ca7bbb

• chore(features): add docstrings

  davidlougheed

  committed
  77cb8df

  Copy full SHA for 77cb8df

• refact(features): factor out iter_features closure

  davidlougheed

  committed
  df2de43

  Copy full SHA for df2de43

• chore: update dependencies

  davidlougheed

  committed
  12630a0

  Copy full SHA for 12630a0

• lint(features): add comments

  davidlougheed

  committed
  863e9d7

  Copy full SHA for 863e9d7

• chore(features): fall back to none right away w/ gene name as feature name

  davidlougheed

  committed
  8f19204

  Copy full SHA for 8f19204

• fix(db): typo handling feature type filter queries for features

  davidlougheed

  committed
  736e705

  Copy full SHA for 736e705

• test: fix hg38 subset genome contig consistency

  davidlougheed

  committed
  5ae8d38

  Copy full SHA for 5ae8d38

• fix(features): bad handling of gff fetch contig + coord off-by-1

  davidlougheed

  committed
  2f8a8d5

  Copy full SHA for 2f8a8d5

• test: more tests for filtering features

  davidlougheed

  committed
  114a1ff

  Copy full SHA for 114a1ff

Commits on May 16, 2024

• perf: add a maximum feature response length limit

  davidlougheed

  committed
  33b630e

  Copy full SHA for 33b630e

• refact: move running tasks to error on startup instead of db init

  davidlougheed

  committed
  0c7276e

  Copy full SHA for 0c7276e

• fix: bad query for tasks querying

  davidlougheed

  committed
  0d76715

  Copy full SHA for 0d76715

• feat: add name_q argument for text searching feature names

  davidlougheed

  committed
  1121e4d

  Copy full SHA for 1121e4d

• fix(features): bad log for feature attribute values

  davidlougheed

  committed
  7a18f65

  Copy full SHA for 7a18f65

• test: task list route

  davidlougheed

  committed
  864f267

  Copy full SHA for 864f267

• lint(db): rm unused function

  davidlougheed

  committed
  3204c07

  Copy full SHA for 3204c07

Commits on May 17, 2024

• feat: include time taken in features query response

  davidlougheed

  committed
  01ae03f

  Copy full SHA for 01ae03f

• chore(features): shorter logs for fallback to feature name/missing ID

  davidlougheed

  committed
  b575d44

  Copy full SHA for b575d44

• fix: lifespan function for moving tasks

  davidlougheed

  committed
  c772412

  Copy full SHA for c772412

• test: tasks detail endpoint

  davidlougheed

  committed
  1425399

  Copy full SHA for 1425399

• feat: fuzzy search params for feature search args: q/name

  davidlougheed

  committed
  b73f11a

  Copy full SHA for b73f11a

Commits on May 21, 2024

• refact(features): simplify feature generator a bit

  davidlougheed

  committed
  fcbe4fa

  Copy full SHA for fcbe4fa

• refact: rewrite feature ingestion to be a bit more RESTy

  davidlougheed

  committed
  b7b8f68

  Copy full SHA for b7b8f68

Commits on May 22, 2024

• chore(streaming): add additional debug log when starting to stream URL

  davidlougheed

  committed
  93699a5

  Copy full SHA for 93699a5

• fix: typo in fasta ref WDL

  davidlougheed

  committed
  24b341d

  Copy full SHA for 24b341d

• chore(config): increase default file response chunk size

  davidlougheed

  committed
  de199e3

  Copy full SHA for de199e3

• fix(workflows): another typo in fasta_ref

  davidlougheed

  committed
  3320bbf

  Copy full SHA for 3320bbf

• fix: feature ingestion

  davidlougheed

  committed
  77b9e63

  Copy full SHA for 77b9e63