Error and warning messages in M-LED & OnSSET

[Alemayehu-ETH]

Hi, I received error and warning messages while running [MLED_hourly.r ] for Rwanda

Error in getGlobalsAndPackages(expr, envir = envir, globals = globals) :
The total size of the 4 globals exported for future expression (‘FUN()’) is 1.01 GiB.. This exceeds the maximum allowed size of 850.00 MiB (option 'future.globals.maxSize'). The three largest globals are ‘FUN’ (998.88 MiB of class ‘function’), ‘clusters_voronoi’ (30.31 MiB of class ‘list’) and ‘exact_extract’ (1.35 MiB of class ‘function’) --- Is this size fixed in the backend.r? A higher size also required for Ethiopia

Warning messages
1: In eval(family$initialize) : non-integer #successes in a binomial glm!
2: st_centroid assumes attributes are constant over geometries
3: In st_centroid.sfc(st_geometry(x), of_largest_polygon = of_largest_polygon) :
st_centroid does not give correct centroids for longitude/latitude data
4: attribute variables are assumed to be spatially constant throughout all geometries
5: In CPL_crs_from_input(x) :
GDAL Message 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with a non-EPSG compliant axis order.
6: In .local(x, y, ...) : Polygons transformed to raster CRS (EPSG:NA) --- repeated many times up to number 50

[giacfalk]

Hi @Alemayehu-ETH, I believe this is a memory error. Could you please try following this?

https://github.com/iiasa/RE4AFAGRI_platform/wiki/M-LED#frequent-issues-and-faqs

Also, try this before running the code:

options(future.globals.maxSize= 891289600)

The additional warnings can be safely ignored.

[Alemayehu-ETH]

Thanks @giacfalk the memory issue is resolved now, and the model is successfully run for Rwanda but received another error while running for Ethiopia from the electricity access module:

Error in if (!is.na(unique(values(resources)))) { :
the condition has length > 1

Regards

[giacfalk]

Thanks, @Alemayehu-ETH . Since the Ethiopia model is a new custom version, better to interact directly among us to check what could be wrong in there!

[Alemayehu-ETH]

Thank you @giacfalk for the support and I will send you a message if the error persists

[Alemayehu-ETH]

Hi Giacomo & Gregory, is it mandatory to use the cluster file from MLED when preparing the datasets for a new country analysis in OnSSET? The cluster dataset from MLED is very large for Ethiopia (around 1.7m features) and it's resulting a memory error when using the GIS extraction notebook to generate the CSV file. I know this part was not covered in the training but want to know whether to use clusters from MLED or generate another cluster using OnSSET method. Thanks, Alemayehu

[giacfalk]

@GregoryIreland could you please have a look at the issue of @Alemayehu-ETH and try to help him?

[GregoryIreland]

Hi @Alemayehu-ETH.
What we typically do in this situation is try to reduce the cluster dataset by removing some of the tiny clusters from the analysis (often which only have a single building in them and less than 10 people - which would also almost always get solar systems as a solution from OnSSET). I wouldn't recommend using the original OnSSET cluster creation process at this point since I believe the current version still uses the Facebook "High Resolution Settlement Layer" for the population estimates which I would say is now quite out of date and the result also gives a very unnatural blocky look to the clusters. You could use an updated raster that would give you better accuracy for the population totals and overall building detection but the clusters would still look bad. They have already done some work to improve this but I don't think it's publically available just yet.

To reduce the size of the MLED clusters (based on GRID3 settlement extents) I would recommend trying a threshold population value of 5 or 10 people and filtering out (removing) the clusters with less than that number and seeing the effect on the number of records and how many people are removed. This can often remove a large number of clusters but only remove a comparatively very small share of the population.

Just note that this could make some of your calibration totals different. e.g. slightly different total population numbers, urbanisation rates, electrification rates etc. Depending on how many people/clusters are removed this might either require you to re-run the MLED code with different/adjusted calibration totals or change the calibration totals in OnSSET appropriately. If it's a very small amount of total population that are removed (about 2%) then many researchers simply adjust/recalibrate the population numbers in the rest of the clusters to match whichever "known" national total you're using/prefer (e.g. UN, or national statistical office value etc)

Also, there isn't really an objective way to determine if the trade-off you end up with is "correct". However, in a previous study where I looked at this in more detail for the DRC and tested different values I was satisfied with the result of reducing the cluster dataset of the DRC by 43% (from about 780k down to ~450k) by removing clusters that only had a single building in them and this only removed 1.5% of the total population from the cluster dataset, which we "reallocated" proportionately to the remaining clusters to reach the original "target" population total. I suspect that a similar result could be possible for Ethiopia.

[Alemayehu-ETH]

Thank you @giacfalk for the heads-up and @GregoryIreland for the detailed explanations. I also prefer to use the cluster from MLED as it captures more areas that are excluded in the OnSSET method. The first cluster was created with <5 population and it only removed <1% of the population. I run the cluster again with <10 population, removing one-third of the clusters and 3.8% of the population. This figure looks fine considering the national electrification plan projected that in the long-term at least 4% of the population can only be served with off-grid technologies. I will re-run the MLED model and get back to you if there are any issues. Thanks, Alemayehu