"Equal" attribution

[matthewlchambers]

A clarification regarding attributing flows equally across multiple industries (used as a fallback method when other attribution sources are not available). If a flow of 6 is to be equally attributed to the NAICS codes 111001, 111002, 111003, and 211000, what is the desired final attribution?

• 111001: 1, 111002: 1, 111003: 1, and 211000: 3
• 111001: 1.5, 111002: 1.5, 111003: 1.5, and 211000: 1.5

I think the first attribution is preferable, but I'm not sure. Thoughts?

[catherinebirney]

My preference is also option 1 - it makes sense to equally attribute a parent naics to child naics, so equally attribute the hypothetical parent of 11 and 21 to 11 and 21 before further attributing to NAICS6. I initially attempted to equally attribute data as you've presented in option 1. However, some of the data we import into flowsa are presented in odd combinations of NAICS levels that I couldn't quite make option 1 work for. My function returned a dataset where the parent and summed child NAICS wouldn't be equal - so I defaulted to option 2. Take a look at EIA_MECS_Energy - that is the dataset that caused problems for me.

[matthewlchambers]

Yes, I noticed that EIA_MECS_Energy was weird that way. But I wrote a pair of functions (in the course of adapting GHG_national_2016_m1 to the recursive .yaml format) that I believe solve the issue:

flowsa/flowsa/data_source_scripts/temp_data_source_functions.py

Lines 109 to 230 in 1605369

	def clean_mecs_energy_fba(fba: FlowByActivity, **_) -> FlowByActivity:
	'''
	This function helps address the unique structure of the EIA MECS dataset.
	The MECS dataset contains rows at various levels of aggregation between
	NAICS-3 and NAICS-6 (inclusive). Each aggregated row contains the total
	for that level of aggregation, even if data are also reported for a less
	aggregated subset of those industries. For example:
	ActivityConsumedBy | FlowAmount | ...
	-------------------------------------
	311 | 110 |
	3112 | 65 |
	311221 | 55 |
	where the 110 reported for 311 includes the 65 reported for 3112, which
	includes the 55 reported for 211221. If we do not address this issue, there
	will be double counting. Additionally, if we are trying to disaggregate
	to the NAICS-6 level, all three rows shown above will be mapped to NAICS-6
	311221 (with the first wo rows also being mapped to several other NAICS-6
	codes as well). We will then over attribute the (double-counted) flows to
	those industries and groups of industries for which more specific detail
	is provided.
	This function addresses the double counting issue. For each aggregated
	industry group, all descendant (less aggregated) industries or industry
	groups for which detailed information is given are subtracted from the
	aggregated total. Using the example from above:
	ActivityConsumedBy | FlowAmount | ...
	-------------------------------------
	311 | 45 |
	3112 | 10 |
	311221 | 55 |
	Additionally, this function adds a column called "descendants", which for
	each industry holds all the descendant industries or industry groups that
	have detailed information provided in the dataset. After mapping to
	industries, but before attribution is performed, this column is used by the
	clean_mapped_mecs_energy_fba function to drop any row that is mapped
	from an aggregated industry group to a less aggregated industry or industry
	group THAT HAS DETAILED INFORMATION GIVEN IN THE MECS (and therefore has
	its own row already) to avoid the over-attribution issue.
	Again using the previous example:
	ActivityConsumedBy | FlowAmount | descendants | ...
	---------------------------------------------------
	311 | 45 | 3112 311221 |
	3112 | 10 | 311221 |
	311221 | 55 | |
	Note that this function is not useful if the desired aggregation level is
	NAICS-2. In such a case, the MECS dataset can be filtered to include only
	the rows with ActivityConsumedBy == "31-33", then disaggregated to 31, 32,
	33 using another dataset (such as the QCEW).
	'''
	mecs = (
	fba
	.query('ActivityConsumedBy != "31-33"')
	.assign(descendants='')
	)
	for level in [5, 4, 3]:
	descendants = (
	mecs
	.drop(columns='descendants')
	.query(f'ActivityConsumedBy.str.len() > {level}')
	.assign(
	parent=lambda x: x.ActivityConsumedBy.str.slice(stop=level)
	)
	.groupby(['Flowable', 'Location', 'parent'])
	.agg({'FlowAmount': 'sum', 'ActivityConsumedBy': ' '.join})
	.reset_index()
	.rename(columns={'ActivityConsumedBy': 'descendants',
	'FlowAmount': 'descendant_flows',
	'parent': 'ActivityConsumedBy'})
	)
	mecs = (
	mecs
	.merge(descendants,
	how='left',
	on=['Flowable', 'Location', 'ActivityConsumedBy'],
	suffixes=(None, '_y'))
	.fillna({'descendant_flows': 0, 'descendants_y': ''})
	.assign(
	descendants=lambda x: x.descendants.mask(x.descendants == '',
	x.descendants_y),
	FlowAmount=lambda x: (x.FlowAmount - x.descendant_flows).mask(
	x.FlowAmount - x.descendant_flows < 0, 0)
	)
	.drop(columns=['descendant_flows', 'descendants_y'])
	)
	return mecs
	def clean_mapped_mecs_energy_fba(fba: FlowByActivity, **_) -> FlowByActivity:
	'''
	This function finishes handling the over-attribution issue described in
	the documentation for clean_mecs_energy_fba by dropping any row in the
	MECS dataset which has been mapped to an industry or industry group which
	is a subset (strict or otherwise) of an industry group listed in the
	descendants columns. So, if 311 and 3112 both appear in the MECS datset,
	3112 will be listed as a descendant of 311 and this function will therefore
	drop a row mapping 311 to 311221 (since more detailed information on 3112,
	which contains 311221, is provided). If 31122 and 311221 do not appear in
	the dataset, a row mapping 3112 to 311221 will not be dropped, since no
	more detailed information on 311221 is given. Further attribution/
	disaggregation should be done using another datatset such as the QCEW.
	'''
	mecs = (
	fba
	.assign(to_keep=fba.apply(
	lambda x: not any([x.SectorConsumedBy.startswith(d)
	for d in x.descendants.split()]),
	axis='columns'
	))
	.query('to_keep')
	.drop(columns=['descendants', 'to_keep'])
	)
	return mecs

since I prefer option 1 as well. This does cause some slight discrepancies in attribution between FBS's generated using the new code versus the old code. Are you comfortable with me ignoring those discrepancies, and continuing to have equal attribution proceed as in option 1?

[catherinebirney]

Oh fantastic - yes, we should use option 1 and ignore any discrepancies caused by the change in methodology. We will note the change in the new github release

[catherinebirney]

@matthewlchambers I'm looping back to this discussion as there is a need to equally attribute census of ag data from parent to child, where the child data isn't published (data contains NAICS4/5 and we want NAICS6). We don't have a general equal allocation function in the recursive branch yet, do we? I see the function to estimate suppressed data for qcew and the functions for the unique situation that is MECS.

[matthewlchambers]

The general equal allocation function is equal_attribution() in flowby.py. It assumes that mapping to sectors/industries has already occurred, then implements the algorithm discussed above.

[catherinebirney]

ah great, thanks!