Proposal of database schema changes for Deps

[AlejandroEsquivel]

Just noting here that this will involve changes to the DB schema. We'll need to have a broader discussion about how we're storing electron metadata in the DB and object store.

from #661 (comment)

[cjao]

Should this wait until after the initial implementation of the new schema? See #654.

[AlejandroEsquivel]

We shouldn't have to wait until implementation but rather we should chat with @FyzHsn so we're on the same page re: schema updates to support deps so we can parallelize work better this sprint. We can chat about this in focus meeting today

[cjao]

Resolution from today's meeting: Faiyaz will implement v9 of the schema.

[santoshkumarradha]

@cjao @FyzHsn can you link the final scheme in this discussion as separate comment once done ? TIA !

[santoshkumarradha]

@mshkanth / @Prasy12 this should give you the details of where this is headed.

[AlejandroEsquivel]

The initial proposal is to extend db schema to have additional columns in the Electron table storing a reference to dependency files used by pipsuch as a requirements.txt. This file can be passed by filepath by the user and/or constructed by us when a list of modules (and corresponding relational operators supplied) e.g the implicit version being:

@ct.electron(
 import_deps={
  "pip": [Path('/requirements.txt'), 'cova==0.102.1rc0']
 }
)

The above should construct a final requirements.txt file which will be uploaded for storage and a reference stored in the database.

Whereas bash dependencies could be stored as a JSON datatype in one column (JSON array).

cc. @cjao @wjcunningham7 @santoshkumarradha