Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EIA923M: Add data maturity flag and 2023 quarterly data #2930

Closed
Tracked by #2902
aesharpe opened this issue Oct 10, 2023 · 2 comments · Fixed by #2936
Closed
Tracked by #2902

EIA923M: Add data maturity flag and 2023 quarterly data #2930

aesharpe opened this issue Oct 10, 2023 · 2 comments · Fixed by #2936
Assignees
Labels
eia923 Anything having to do with EIA Form 923 rmi

Comments

@aesharpe
Copy link
Member

aesharpe commented Oct 10, 2023

Our current 923 archiver contains annual and YTD data. We need to come up with a way to distinguish the data maturity of each of the partitions. The way we currently determine data_maturity is via the add_data_maturity() function in the extract/excel.py module. In the future it might make sense to add a data_maturity field to the archive metadata, but for now, we'll update the add_data_maturity() to account for the EIA923 data. There's a category for the data_maturity column called incremental_ytd that was created for the purpose of this type of data.

The raw files are formatted as such (examples):

EIA923_Schedules_2_3_4_5_M_07_2023_20SEP2023.xlsx
EIA923_Schedules_2_3_4_5_M_12_2022_Early_Release.xlsx

Where the value after M is the the last month the have data. Data with 12 is annual, data with values less than 12 are YTD. We can use regex to extract the month from here and add the incremental_ytd flag if it's not 12.

This issue will also inherently add the 2023 quarterly data for 923.

@aesharpe aesharpe changed the title EIA923M: Monthly dataset provided through the same link every month. Add a method to distinguish the monthly file from the annual ones and parse date from website (file name has date of upload, not date of data). EIA923M: distinguish the monthly file from the annual ones Oct 10, 2023
@aesharpe aesharpe changed the title EIA923M: distinguish the monthly file from the annual ones EIA923M: distinguish the monthly from the annual filings Oct 10, 2023
@aesharpe aesharpe changed the title EIA923M: distinguish the monthly from the annual filings EIA923M: Add data maturity to EIA923 monthly data Oct 10, 2023
@aesharpe aesharpe changed the title EIA923M: Add data maturity to EIA923 monthly data EIA923M: Add data maturity flag and 2023 quarterly data Oct 11, 2023
@aesharpe aesharpe moved this from New to In progress in Catalyst Megaproject Oct 12, 2023
@aesharpe aesharpe self-assigned this Oct 12, 2023
@aesharpe aesharpe added eia923 Anything having to do with EIA Form 923 rmi labels Oct 12, 2023
@aesharpe aesharpe linked a pull request Oct 12, 2023 that will close this issue
@cmgosnell cmgosnell moved this from In progress to In review in Catalyst Megaproject Oct 30, 2023
@aesharpe
Copy link
Member Author

aesharpe commented Nov 2, 2023

Close with #2936

@aesharpe aesharpe moved this from In review to Done in Catalyst Megaproject Nov 2, 2023
@e-belfer
Copy link
Member

Marked done but not closed, closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
eia923 Anything having to do with EIA Form 923 rmi
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants