Lighten the Load: A new data store for eLCI

[dt-woods]

It goes without saying that the ElectricityLCI repository is getting big (>250 MB). This slows down pulls/pushes and makes the repository less user-friendly. The reliance on the repository for storing data (e.g., public life cycle inventories), intermediate data files (e.g., StEWI FRS bridge files), and templates, is not sustainable as the repository is only expected to grow, as new LCIs become available (e.g., 2020 natural gas and new nuclear).

A better handling of large data files (e.g., >100 kB) would be to host them on a public data warehouse, where their metadata, version/history, citation, and document summaries can be expressed for traceability and transparency, which are both lacking/difficult in the current organization of the repo.

[dt-woods]

Can these StEWI FRS bridge files and stewicombo files go?

• eGRID_2014_TRI_2014_NEI_2014_RCRAInfo_2013_fromstewicombo.csv
• eGRID_2016_TRI_2016_NEI_2016_RCRAInfo_2015_fromstewicombo.csv
• eGRID_2016_fromstewicombo.csv

[dt-woods]

Is this template file still valid?

• template.xlsx

[bl-young]

Is this template file still valid?

• template.xlsx

No that is not in use.

[bl-young]

I do agree that hosting files of any significant size should happen externally. I've also wondered if it would make sense to split out the upstream data and related processing into a separate package, like ELCI_upstream or something to streamline/simplify things a bit.

Unfortunately, we've found that deleting large files doesn't remove them from the git history and so does not impact the size of the repository (an issue we've faced with fedelemflowlist). Something like this might be necessary, but I have not tried it: https://rtyley.github.io/bfg-repo-cleaner/#usage

[dt-woods]

The quick and the easy is to delete the old repository, which has bloated, and create a nice new clean repo, which you don't dump data in to.

[WesIngwersen]

Can these StEWI FRS bridge files and stewicombo files go?

• eGRID_2014_TRI_2014_NEI_2014_RCRAInfo_2013_fromstewicombo.csv
• eGRID_2016_TRI_2016_NEI_2016_RCRAInfo_2015_fromstewicombo.csv
• eGRID_2016_fromstewicombo.csv

I agree with moving all these static files to a data server. We can put all stewi-based outputs on our Data Commons stewi folder