20231009 developer call notes

[tomwhite]

20231009

Pre-notes

• Notes from previous meeting

PRs

• sgkit#1129 Fix memory leak (see cyvcf2 # 280)
• sgkit#1132 Skip parts of a VCF that have already been done

Issues

• sgkit#1133 Use Xarray's map_blocks rather than Dask's
• sgkit#1134 Mergify has stopped working

Discussions

• Publication

Notes

Attendees

1. Tom White
2. Jerome Kelleher
3. Ben Jeffery

Discussion

• sgkit#1129 Fix memory leak (see cyvcf2 # 280)
• sgkit#1132 Skip parts of a VCF that have already been done
  • JK: Computation finished in less than 2 weeks. But don’t know how long! (Away on holiday.)
  • TW: Two memory problems?
  • JK: Should both be fixed in cyvcf2
  • JK: Haven’t run with latest cyvcf2 release. But it should fix it.
  • JK: Non-recovery from Dask is bigger problem.
  • BJ: Did a quick hack. Not a Dask bug, as it doesn’t know you are running it for side effects. Discussion on Dask ticket. Worker memory is still growing over time.
  • BJ: Will ask Patrick H from Coiled when he visits if there is an API to say “run tasks for side effects” in Dask.
  • JK: Now running benchmarks for the publication. We should include SAV on side plot.
  • TW: I spoke to Jacob Tomlinson at PyCon UK and he said GPUs are a good fit for CSV parsing. Would need CUDA C++ code, or Numba (but Numba doesn't have good support for strings?)
• sgkit#1133 Use Xarray's map_blocks rather than Dask's
  • TW: This is motivated by Cubed.
  • JK: Is this a simple change?
  • TW: Probably not as the APIs are different. So will need some testing.
• sgkit#1134 Mergify has stopped working
  • TW: Fixed