Cumulative imputation links are incorrect for FIC

[robertswh]

The imputation_group column is created by looking for changes in three columns:

• missing_value which is True or False depending on a return or non-return
• group
• reference

This works for most cases, but not for FIC.

The imputation_group column should look at imputation_marker for changes instead of missing_value. There is some pre-processing needed because we look for changes in numerical columns, not categorical. One solution could be to create a new column (e.g. mapped_marker) for mapping of markers to numbers, e.g. mapping = {"r": 1, "c": 2, "fir": 3, "bir": 4, "fic": 5} (exact numbers doesn't matter as long as they're unique), then the condition would be (dataframe["mapped_marker"].diff(time_difference) != 0).

This condition would replace the following condition:

monthly-business-survey-results/src/cumulative_imputation_links.py

Line 49 in d2e8a38

(dataframe["missing_value"].diff(time_difference) != 0)

[AntonZogk]

By using this condition
(dataframe['imputation_marker'].ne(dataframe['imputation_marker'].shift().bfill()).astype(int) != 0)

instead of

monthly-business-survey-results/src/cumulative_imputation_links.py

Line 49 in d2e8a38

(dataframe["missing_value"].diff(time_difference) != 0)

fixed the issue for cummulative links , it returns the correct cumulative link. But still the imputed value afterwards is wrong.

[AntonZogk]

When calculating the FIC the create_and_merge_imputation_values function should point at Imputed_value instead of constructed since it has all the constructed values and we can't do the required forward fill

monthly-business-survey-results/src/apply_imputation_link.py

Line 85 in d2e8a38

"fill_column": "constructed",

[AntonZogk]

monthly-business-survey-results/src/apply_imputation_link.py

Lines 55 to 88 in d2e8a38

	imputation_config = {
	"c": {
	"intermediate_column": "constructed",
	"marker": "C",
	# doesn't actually apply a fill so can be forward or back
	"fill_column": auxiliary,
	"fill_method": "ffill",
	"link_column": construction_link,
	},
	"fir": {
	"intermediate_column": "fir",
	"marker": "FIR",
	"fill_column": target,
	"fill_method": "ffill",
	"link_column": cumulative_forward_link,
	},
	"bir": {
	"intermediate_column": "bir",
	"marker": "BIR",
	"fill_column": target,
	"fill_method": "bfill",
	"link_column": cumulative_backward_link,
	},
	"fic": {
	# FIC only works if the C is in the first period of the business being
	# sampled. This is fine for automatic imputation, but should be careful
	# if manual construction imputation is done
	"intermediate_column": "fic",
	"marker": "FIC",
	# this has to have the same name as the intermediate column for constructed
	"fill_column": "constructed",
	"fill_method": "ffill",
	"link_column": cumulative_forward_link,
	},

The marker values of the above must match (case sensitive) these:

monthly-business-survey-results/src/imputation_flags.py

Lines 120 to 125 in d2e8a38

	"r": df["r_flag"],
	"fir": ~df["r_flag"] & df["fir_flag"],
	"bir": ~df["r_flag"] & ~df["fir_flag"] & df["bir_flag"],
	"fic": ~df["r_flag"] & ~df["fir_flag"] & ~df["bir_flag"] & df["fic_flag"],
	"c": ~df["r_flag"]
	& ~df["fir_flag"]