Skip to content

ODC SC Meeting 2019 07 25

Jump to bottom Edit New page
gamedaygeorge edited this page Jul 24, 2019 · 15 revisions

Attendees

  • Andrew Cherry (Chair)
  • Kirill Kouzoubov (GA)
  • Alex Leith (GA)
  • Luigi Di Fraia (Catapult)
  • Peter Wang (Data61)
  • Rob Woodcock (CSIRO)
  • Syed Rizvi (AMA)
  • Tony Butzer (USGS)
  • George Dyke (Symbios)
  • Chris Morgan (Frontiersi)

Apologies

Minutes

See Google Doc.

Agenda

  • Welcome
  • Previous Actions
    • See below
  • Enhancement Proposals
  • State of the Cube Highlights
    • CSIRO
      • All of the CSIRO staff have taken significant leave during this period so a smaller number of outcomes this month
      • CSIRO Data pipelines (line ingester) and ODC/DEA indexing components have been successfully run on the CSIRO AWS EASI EO hub deployment:
        • DEA indexes had a dozen or so containers fail during indexing. No logs were returned, the pods just hung, with s3-find and some other processes just spinning small amounts of CPU time. The (vast majority) of other pods ran to completion successfully.
        • CSIRO data pipelines successfully ordered data from multiple global source archives and indexed/ingested into the CSIRO environment for small scale tests.
        • Next step: Scale up the CSIRO data pipelines robots in a manner similar to the DEA indexing approach (k8s jobs) and hopefully debug the DEA indexing issue
      • The CSIRO AWS EASI EO hub deployment has gained interest from a group of Pangeo connected folks in CSIRO Oceans and Atmosphere. A demonstration of the capability was provided and we are now looking at join development opportunity. The Pangeo stack, as an ODC would expect is basically the same as the ODC one but with additional emphasis on netcdf support.
        • One issue exists with holoviews - holoviews is currently crashing the jupyter notebook kernel for unknown reasons. It's probably an issue with library versions but if anyone has a working ODC with holoview (or more specifically geoviews) running we'd a appreciate a look at your image build so we can compare the library and OS package versions. The O&A folks normally use Conda to manage this but ODC is using pip so its a little more convoluted to resolve. - CSIRO EASI EO k8s environment was updated to support multiple-instance types when scheduling SPOT nodes, significantly increasing the SPOT node availability. PR has been posted to dataube-k8s-eks.
    • Catapult
      • TBA
    • GA
      • TBA
    • USGS
      • USGS is focused on delivery data over Africa to prototype Collection 2 - Level 2
      • GA will then index this data into an Open Data Cube and look at continental scale applications in the cloud
      • Tony Butzer and Randy Sunne will be largely offline for Steering Council activities during this very large data load operation
    • AMA
      • TBA
    • FrontierSI
      • Working with AMA setting up ODC Cluster
      • Working with GA on DEA sandbox
  • Other Business
    • TBA
  • Next Meeting and Close
    • Date

Actions

See the action tracking project.

Outstanding Actions

New Actions

Welcome to the Open Data Cube

Home

Clone this wiki locally