Add config across scripts and overall pipeline script

[sgreenbury]

Contributes towards #44.

[Hussein-Mahfouz]

Hey @sgreenbury , I wanted to add some functionality in the config file but didn't want to conflict with your workflow / wanted to understand how the functions are working first.

When using travel_times matrix, I want to check if it exists, and if it doesn't create one. I've implemented the logic here:

acbm/scripts/3.1_assign_primary_feasible_zones.py

Lines 75 to 107 in 771edff

	# TODO: check config file to see if we have a travel time matrix. If not, create one
	travel_times = pd.read_parquet(
	acbm.root_path / "data/external/travel_times/oa/travel_time_matrix.parquet"
	)
	logger.info("Travel time matrix loaded")
	# # Load the configuration file
	# config_file_path = 'config.toml'
	# travel_time_matrix_path = acbm.root_path / "data/external/travel_times/oa/travel_time_matrix.parquet"
	# with open(config_file_path, 'r') as config_file:
	# config = toml.load(config_file)
	# # Check if travel_times is set to true and try to load the travel time matrix
	# if config.get('travel_times') == True:
	# try:
	# travel_times = pd.read_parquet(travel_time_matrix_path)
	# print("Travel time matrix loaded successfully.")
	# except Exception as e:
	# logger.info(f"Failed to load travel time matrix: {e}")
	# logger.info("Creating a new travel time matrix.")
	# # Create a new travel time matrix based on distances between zones
	# travel_times = zones_to_time_matrix(zones = boundaries, id_col = "OA21CD")
	# # TODO: save the travel time matrix
	# # If travel_times is not true or loading failed, create a new travel time matrix
	# if config.get('travel_times') != True:
	# logger.info("No travel time matrix found. Creating a new travel time matrix.")
	# # Create a new travel time matrix based on distances between zones
	# travel_times = zones_to_time_matrix(zones = boundaries, id_col = "OA21CD")
	# logger.info("Travel time estimates created")

Could we add that to the config? We can discuss Friday. Related to #48

[sgreenbury]

Could we add that to the config? We can discuss Friday. Related to #48

Looks good to me adding like that to the config and then config["parameters"]["travel_times"] to get the value if it is with the other config parameters we've added so far. I wonder if it needs to be configurable though since this block might always be worth running? Or would there be runs where we do not want to load the matrix if it exists?

[Hussein-Mahfouz]

I think the important thing is to put TRUE or FALSE in the config depending on whether we have a travel time matrix. That would then be used here. The block you refer to just handles the case where travel_times is accidentally set to TRUE but there is no actual travel_times parquet file. Maybe not necessary.

Somewhat related, I am thinking of using pandera to check the schemas of all dfs before applying functions.

[sgreenbury]

Following first full run for Leeds, as discussed with @Hussein-Mahfouz and @BZ-BowenZhang merging this PR.