Hierarchical Matching

[DanRunfola]

I'm moving some of our conversations about hierarchical matching out of issues and into this discussion area.

Essentially, I want to get a pulse on different strategies we could use for hierarchical standardization (ensuring ADM2 nests in ADM1, ADM3 in ADM2, and so on), and what the community might ultimately prefer.

Just to get started:
#2326 (comment)

[chapmanjacobd]

https://github.com/wmgeolab/geoBoundaryBot/blob/78294c59a91456f0e79fb13b30ca9d0400ad0136/gbCGAZBuild.py#L182

this could be extended to include ADM3..5

alternatively,

qgis_command = (
    "env QT_QPA_PLATFORM=offscreen qgis_process run native:joinattributesbylocation --"
    " METHOD=2 PREDICATE=0 DISCARD_NONMATCHING=0"
   f" INPUT='{temp_file}|layername=g{adm}'"
   f" JOIN='{previous_file}|layername=g{adm - 1}' JOIN_FIELDS='shapeName;shapeGroup;id;longName' PREFIX='g{adm - 1}_'"
   f""" OUTPUT='ogr:dbname="{out_file}" table="g{adm}" (geom)' """
)

[tfardet]

Just to check is this referring to what is done for instance in GADM, where each level 4 (for instance) entry has the name of the areas that contain it at levels 3, 2, and 1?
If so, I would propose to do this as in the GeoNames database, where each entry has adminX (X in {1, 2, 3, 4}) values that let you know what are its unique code and the codes of the higher admin boundaries that contain it.
I would propose geocodes (as in GeoNames) rather than names (as in GADM) because the former are always unique and you don't need to worry about the language.

[DanRunfola]

We haven't made a choice on the details of how we'll implement this - we try to be friendly to our "ESRI" community, so I would want whatever we have to be human-readable as well as machine readable. Immediate thought on this is something like:
A) A column with "names", akin to GADM, which would allow folk to use gB as a drop-in replacement (this is the most frequent critique we get from folk who want to swap from GADM). Format would be very similar, names with dashes or something, that would need to be parsed.
B) A column with a machine-readale hierarchy, with a set of geoBoundaries IDs indicating all boundaries in our database that a given boundary falls into (say, comma delimited). I would think to structure this like a JSON, so that you always have 6 objects - one for ADM0, ADM1, ADM2, ADM3, ADM4, and ADM5. If you were then calculating this field for an ADM3, it would look something like:

[[ADM0gbCode, ADM1gbCode, ADM2gbCode, OwnCode, [List of ADM4s that fall within this unit], [List of ADM5s that fall within this unit]]

Where you would have "None" for cases where there isn't any data on levels below a given case (or above, for that matter).

Any high level thoughts on that strategy? Very up-in-the-air for now!

[tfardet]

OK, having worked with both GADM and GeoNames, I would say that the NAME_1, NAME_2... convention is a lot less practical and slow than the admin1, admin2... geocode entries of the area.

Geocodes are shorter (faster search and comparisons) and often systematic such that you can deduce e.g. admin2 and 3 from admin4.

[[ADM0gbCode, ADM1gbCode, ADM2gbCode, OwnCode, [List of ADM4s that fall within this unit], [List of ADM5s that fall within this unit]]

Ah, I did not expect this; do you mean that this is for descending order? I.e. store the sublevel admins that are contained in an area?
If so I would strongly recommend against this: it would make the files way too large.
In both GADM and GeoNames, this is also not necessary because you can just filter (in your example) the admin4 entries that have the relevant NAME_3 or admin3 entry respectively.

[DanRunfola]

Hey @tfardet - are you referring to the GeoNames geocodes?

We've been working on ways to try and crosswalk between geoboundaries and a wide range of alternative ID schemes - including ISO codes for sub-levels, the UN P-Codes, and a few others. Ultimately, the GeoNames geocodes would be another example of an ID scheme we would eventually want to tie into, but we won't be there immediately.

I agree on the file size issue, going to lower units would be too much.

What I want to avoid is turning into GADM, with so many ancillary columns that a lay user can't figure out what's going on (and, equally importantly, accuracy degradation as we add more elements we're maintaining). For example, for an ADM2, GADM would provide the following columns for each unit:
GID_0 - ISO code of the home country in most cases
COUNTRY - The name of the country the unit falls into
GID_1 - I belive this is the geonames ID of the home country?
NAME_1 - Name of the ADM1 the ADM2 falls in
NL_NAME_1 - I have no idea
GID_2 - I belive geonames ID of ADM2?
NAME_2 - Name of the ADM2
VARNAME_2 - Alternative name of the ADM2
NL_NAME_2 - No idea
TYPE_2 - I am not sure, I think this might be canonical name based on the examples I looked at.
ENGTYPE_2 - English canonical name?
CC_2 - No idea
HASC_2 - Looks like some ID

Contrast this to geoBoundaries, which has the exact same schema across all data products (i.e., no matter what level you grab, you get these columns):
shapeName - The name of the unit
shapeISO - The ISO Code of the unit
shapeID - The geoBoundaries ID
shapeGroup - The ISO of the home country
shapeType - The type (i.e., ADM2)

So the question here is how we could add hierarchical information while also retaining what I would consider the interpretability / usability of gB. While also operating within the constraints of a flat attribute table to accomodate shapefiles / geojsons. And also wanting to give users easy filter capabilities. Which seems an impossible set of capabilities to offer :).

One alternative solution would be to provide hierarchical information as an optional addon join - i.e., at checkout, the user ticks a box (or something), and when they get their copy of geoBoundaries it also provides a csv to join (or, perhaps, just provides an alternative version with the join already done). In that case, we would have more flexibility to add arbitrary columns. I.e., then I could see adding something like (with multiple columns according to the number of parents):

shapeADM1Name - Name of ADM1
shapeADM1ID - gbID of ADM1 parent

Then, if the user checks, say, a "geonames" box (that won't exist for a very long time :) ), they would also get:
ADM2_geoName - geoNameID of ADM2
ADM1_geoName - geoNameID of ADM1

Obviously still at the phase of brainstorming here, so any thoughts welcome.

[chapmanjacobd]

I think this data could exist as a sidecar JSON file.

It could also be used to check if new data products spatially include or exclude a different number of lower level features than the previously committed geoboundaries version