You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Our current 923 archiver contains annual and YTD data. We need to come up with a way to distinguish the data maturity of each of the partitions. The way we currently determine data_maturity is via the add_data_maturity() function in the extract/excel.py module. In the future it might make sense to add a data_maturity field to the archive metadata, but for now, we'll update the add_data_maturity() to account for the EIA923 data. There's a category for the data_maturity column called incremental_ytd that was created for the purpose of this type of data.
Where the value after M is the the last month the have data. Data with 12 is annual, data with values less than 12 are YTD. We can use regex to extract the month from here and add the incremental_ytd flag if it's not 12.
This issue will also inherently add the 2023 quarterly data for 923.
The text was updated successfully, but these errors were encountered:
aesharpe
changed the title
EIA923M: Monthly dataset provided through the same link every month. Add a method to distinguish the monthly file from the annual ones and parse date from website (file name has date of upload, not date of data).
EIA923M: distinguish the monthly file from the annual ones
Oct 10, 2023
aesharpe
changed the title
EIA923M: distinguish the monthly file from the annual ones
EIA923M: distinguish the monthly from the annual filings
Oct 10, 2023
aesharpe
changed the title
EIA923M: distinguish the monthly from the annual filings
EIA923M: Add data maturity to EIA923 monthly data
Oct 10, 2023
aesharpe
changed the title
EIA923M: Add data maturity to EIA923 monthly data
EIA923M: Add data maturity flag and 2023 quarterly data
Oct 11, 2023
Our current 923 archiver contains annual and YTD data. We need to come up with a way to distinguish the data maturity of each of the partitions. The way we currently determine data_maturity is via the
add_data_maturity()
function in theextract/excel.py
module. In the future it might make sense to add adata_maturity
field to the archive metadata, but for now, we'll update theadd_data_maturity()
to account for the EIA923 data. There's a category for thedata_maturity
column calledincremental_ytd
that was created for the purpose of this type of data.The raw files are formatted as such (examples):
EIA923_Schedules_2_3_4_5_M_07_2023_20SEP2023.xlsx
EIA923_Schedules_2_3_4_5_M_12_2022_Early_Release.xlsx
Where the value after
M
is the the last month the have data. Data with12
is annual, data with values less than12
are YTD. We can use regex to extract the month from here and add theincremental_ytd
flag if it's not 12.This issue will also inherently add the 2023 quarterly data for 923.
The text was updated successfully, but these errors were encountered: