Playing around with dynamic specifications from wkt2 strings #2
Conversation
|
weird, so my fix apparently broke the tests because sometimes a None is returned for the authority. I am digging into it now but it looks like converting to a dict first will always resolve the id for example the CRS for Steins (2015) - Sphere / Ocentric has a valid authority of ('IAU_2015', '200286700') but for Toutatis (2015) - Sphere / Ocentric, I get a None for the authority, but when converted to a dict first I get {'authority': 'IAU', 'code': 200417900, 'version': 2015} which seems fine |
|
weird, so here are the two wkts: and I see Id's in both of these but Steins has an anchor defined for the ellipsoid while the first (failing) one doesn't |
|
Those are some legit gymnastics in order to get the codes! Glad you got that figured out. I want to look at TiTiler / Morecantile and determine how best to associate the input data proj with the identifier. For example IAU_30100_2015 is not going to be meaningful to 99.9% of the users. I sent the link to the test TiTiler deploy (AWS) via email. At the /docs URL, you can see the swagger spec/UI. In there, my first approach was to add body specific endpoints (which is not ideal for a myriad of reasons, but works for now). So, |
|
Messing with this a bit more this morning. Here are a few things that I noticed:
import morecantile
tms = morecantile.TileMatrixSet.parse_file('Moon_2000_tms.json')
Option 1: Hack the CRSes in the JSON files.In the short term, it might be most elegant to perform this hack (replace Edit: The above is partially correct. Randomly browsing opengis.net I see an endpoint for IAU! So, the hack is unnecessary in four cases. What needs to be done is to get the correct supportedCRS and boundingBox:crs links for the code. opengis.net has entries for Mars2000, Mercury2000, Moon2000, and Venus2000 (which is kind of random...). If one hacks the {
"type": "TileMatrixSetType",
"title": "Sun (2015) - Sphere / Ocentric",
"identifier": "IAU_1000_2015",
"supportedCRS": "http://www.opengis.net/def/crs/EPSG/0/4326",
"boundingBox": {
"type": "BoundingBoxType",
"crs": "http://www.opengis.net/def/crs/EPSG/0/4326",
"lowerCorner": [
-90.0,
-180.0
],
"upperCorner": [
90.0,
180.0
]
},Option 2: Register a custom TMS to morecantile
from pyproj import CRS
mars_crs = CRS.from_authority("ESRI", 104971)
ESRI104971 = morecantile.TileMatrixSet.custom(
crs=mars_crs,
identifier="Mars_2000_Sphere_ESRI",
extent=mars_crs.area_of_use.bounds,
matrix_scale=[1, 1],
)
iau_mars_crs = CRS.from_authority("IAU", "49900")
IAU49900 = morecantile.TileMatrixSet.custom(
crs=iau_mars_crs,
identifier="Mars_2000_Sphere_IAU",
extent=mars_crs.area_of_use.bounds,
matrix_scale=[1, 1],
)
import urllib
with urllib.request.urlopen('https://raw.githubusercontent.com/pdssp/planet_crs_registry/main/data/result.wkts') as response:
resp = response.read().decode(response.headers.get_content_charset())
import os
geocrss = []
for wkt_str in resp.split(os.linesep + os.linesep):
if 'GEOGCRS' in wkt_str[:7] and 'TRIAXIAL' not in wkt_str: # insanely hacky
geocrss.append(wkt_str)
for crs in geocrss:
crs_obj = CRS(crs)
title = crs_obj.name
auth = crs_obj.to_authority(min_confidence=25)
if auth is not None:
authority_version, code = auth
authority, version = authority_version.split('_')
identifier = f'{authority}_{code}_{version}'
if crs_obj.area_of_use:
bounds = crs_obj.area_of_use.bounds # Not sure that this is working?
else:
bounds = [-180, -90, 180, 90]
custom_tms = morecantile.TileMatrixSet.custom(
crs=crs_obj,
identifier=identifier,
extent=bounds,
matrix_scale=[1, 1])
# Strange append because morecantile.tms is a custom object
morecantile.tms = morecantile.tms.register(custom_tms)
morecantile.tms.list()Results in (truncated): ['LINZAntarticaMapTilegrid',
'EuropeanETRS89_LAEAQuad',
'CanadianNAD83_LCC',
'UPSArcticWGS84Quad',
'NZTM2000',
'NZTM2000Quad',
'UTM31WGS84Quad',
'UPSAntarcticWGS84Quad',
'WorldMercatorWGS84Quad',
'WGS1984Quad',
'WorldCRS84Quad',
'WebMercatorQuad',
'IAU_1000_2015',
'IAU_19900_2015',
'IAU_19901_2015',
'IAU_29900_2015',
'IAU_30100_2015',
'IAU_39900_2015',
'IAU_39901_2015',
'IAU_40100_2015',
'IAU_40200_2015',
'IAU_49900_2015',
...All of this is to say: How are we anticipating one use these custom TMS? |
|
@jlaura That is a lot of great detective work and I can't respond to it all now, but a few things I can say:
|
|
@jlaura this looks relevant to our discussion opengeospatial/NamingAuthority#212 |
|
Good find! This is what Trent has been talking about. I did not realize the timeline and that it would be linking through opengis.net. That solves the problem on our side as far as I can tell! We'll have to check morecantile/proj to ensure that they are encoding the URL properly, e.g., that they didn't hard code the Tangential: Here is where I've been iterating with a STAC proj extension maintainer about getting the codes supported in there. All these small friction points will need either good docs on our side or changes on projects in order to make use of things like the official IAU codes. |
|
as an update to point 2 I made above, I made this PR for morecantile to support generating more-correct scaleDenominators by using the CRS ellipsoid's semi major axis rather than hard-coding the earth radius (developmentseed/morecantile#92). Looking into it the WKSS part may be partially deprecated in TMS 2.0 so we may not need to define additional wkss's I totally agree about figuring out a way to remove NAIF Ids from the names of the bodies, but I am unsure what the right solution would be for that at the moment if the NAIF id is used is the code and if the code must be an integer? for the urls encodings in morecantile this looks like the relevant bit of code |
|
I'll make a pr for that also as it could use the same logic I added to this pr to grab the code/version/authority (edit: done! developmentseed/morecantile#93) |
|
😆 I was just logging on to get that pushed in. 👍 Glad to see the PRs opened. With a hopeful update to opengis, I think that these will be all set to deploy without a bunch of hackiness. Once that is the case, I am happy to host these as a lambda at a USGS URL as a community service. |
|
👋 this is a great conversation 😄 |
|
@vincentsarago thanks so much for that info, good to know about the TMS spec, I had assumed wrongly about TMS 2.0 already being implemented. We don't have an immediate need for 2.0 in morecantile but likely that would be of interest eventually. I am still very much learning about the TMS spec in general, so it would be very helpful to get your guidance on what else needs to be done to support these IAU CRSs better in morecantile and also in projects downstream of it. |
|
@jlaura looks like Vincent merged in the first of the two PRs earlier today, there was a bit more work to do with the 2nd one but I think we're now at a much better spot with it and hope it can be merged in soon. Time to come up with more tasks! |
|
one thing that could be good is to think about the maxzoom parameter a bit. I could imagine that for low res data sets like MOLA we could tune that to lower numbers but we'd want TMS specs usable for HiRISE images also. Maybe it's best to assume a higher zoom level always for each body and assume clients would do the right thing and not attempt to render tiles much finer than the best resolution available. |
|
@AndrewAnnex 🎉 That's great news! I think we need to support a really wide range zoom range and then assume that the client or user will do the correct thing. For example, if we have a TMS MOLA base (resolutions is like 463m or so?) and someone wants to tile a long CTX string (50x lines at 6mpp), I think we want them to be able to use the same TMS endpoints. Next steps on the TMS:
Are you thinking of other next steps for TMS or in general? I am not super fluent in the topography encoding, but you seemed pretty excited about that. Can I assist there? I've been starting to look at vector tiles and SLD symbology definitions as well. I have a pygeoapi deploy as an AWS lambda that I am trying to use to learn about the vector side of things. COG and STAC standards / best practices are probably right up there too. For example, we have WKT2 codes, but they will not properly encode out of GDAL without some coaxing. 😆 So, lots of areas! |
|
@jlaura sounds good regarding the wide zoom ranges, I'll do some thinking to come up with a reasonable limit (possibly 24 or 25; for earth zoom level 24 is 0.009m GSD at the equator assuming an equirectangular projection, so smaller bodies would be even finer scale resolution than that, 24 may be more sensible as that would ensure tiles indexes remain within 24bits if that even matters... some things to think about anyways) Next steps sound good. Once the morecantile prs are merged in (and maybe a new version is released) the changes in this PR can be merged. I sent you an email before I saw this and had some thoughts about the topography encoding which for now I think are somewhat independent from this until the opengis update happens. My biggest unknown is what else needs to happen to js frontends assuming we get these parts working. I also haven't thought much about the vector side yet, interested to hear why the wkt2 codes wouldn't work with ogr/gdal... We may need to move the discussion elsewhere on github, maybe as issues on this repo? |
|
morecantile 3.2.3 is available on pypi https://pypi.org/project/morecantile/3.2.3/ 🥳 |
|
|
||
| for tmsp in crss: | ||
| # create the tms object | ||
| tms = morecantile.TileMatrixSet.custom(**asdict(tmsp)) | ||
| _d = json.loads(tms.json(exclude_none=True)) # get to pretty printed json | ||
| with open(f'./{tmsp.crs.name}_tms.json', 'w') as dst: | ||
| with open(f'./{tmsp.identifier}_tms.json', 'w') as dst: | ||
| # dump to json | ||
| json.dump(_d, dst, indent=4, ensure_ascii=False) | ||
| print(f'wrote {dst.name}') |
There was a problem hiding this comment.
So this generate script will create a lot of TMS right? I'm kinda worry now that when importing planetcantile we might try to parse too many JSON files before being able to do any operation.
planetcantile/planetcantile/defaults.py
Line 10 in 5ea2577
Maybe this could be fixed on morecantile side where we only parse the JSON file when we need it 🤷
There was a problem hiding this comment.
@vincentsarago Correct - a whole bunch of JSON files!
A change to morecantile would be a pretty big change to the logic in morecantile.defaults right? Looks like that is currently globing, loading, and ultimately creating a TileMatrixSet object that is used up stream (e.g., by titiler). Is that correct?
If so, we'd want to make sure that users of Morecantile (and upstream) can see the available TMSs without having to parse all the JSON.
- We could enforce filename mapping to IDs and just read the filenames (this has issues).
- We could read just a portion of the JSON (this is hacky and has other issues including a giant WHY).
- We could read all of them using a library like msgspec which is supposed to be significantly faster. (Caveat, we have lots of file handles to grab for small JSON files, so this would need benchmarking first. Could we go to a single TMS file with an array of TMS objects?).
- If we could advertise properly, we could read on use and then keep that TMS in memory, so cache.
I guess, I would really want to make sure that we can advertise the availability of the TMS while still maintain good performance? Maybe I'm worried about the wrong thing though!
There was a problem hiding this comment.
I did start a PR to show how it could be developmentseed/morecantile#94
I don't think this is too much of a change. in planetcantile it will look like:
import importlib.resources as importlib_resources
from morecantile.defaults import default_tms, TileMatrixSets
planet_tms_jsons = importlib_resources.files('planetcantile.data').glob('*.json')
default_tms.update(
{p.stem: p for p in planet_tms_jsons}
)
tms = TileMatrixSets(copy(default_tms))
# Or to only use TMS from planetcantile
import importlib.resources as importlib_resources
from morecantile import TileMatrixSets
planet_tms_jsons = importlib_resources.files('planetcantile.data').glob('*.json')
default_tms = {p.stem: p for p in planet_tms_jsons}
tms = TileMatrixSets(copy(default_tms))The other solution would be to simply use TILEMATRIXSET_DIRECTORY environment variable to tell morecantile to look for json document within a specific directory. https://github.com/developmentseed/morecantile/blob/main/morecantile/defaults.py#L16-L18
There was a problem hiding this comment.
We could enforce filename mapping to IDs and just read the filenames (this has issues).
Well right now morecantile assume this 🙈
I guess, I would really want to make sure that we can advertise the availability of the TMS while still maintain good performance? Maybe I'm worried about the wrong thing though!
Right now (with the PR) we keep the advertising part (but using the file name).
There was a problem hiding this comment.
Well right now morecantile assume this 🙈
😆 This is good! And we can use the environment variable on the titiler side to offer these via our lambda deploy. I might hack the check to opengis.net out of a version that I have deployed (only to be able to test until they update) and then get that URL shared to you both for verification.
Knowing about that env variable and having the code snippet is AWESOME! 🌟
There was a problem hiding this comment.
I am worried about projects that use morecantile's default tms set without allowing overrides. Yes it's possible to make PRs against them to accept a tms set as a parameter but my suggestion could be a very cool mechanism to support
There was a problem hiding this comment.
oh something I missed @vincentsarago, I have a code snippet like that already in planetcantile in https://github.com/AndrewAnnex/planetcantile/blob/579780ac2ecdcb787e38aefe6255adf136923f3f/planetcantile/defaults.py
There was a problem hiding this comment.
err "like" it but not exactly it, also not working currently
There was a problem hiding this comment.
I am worried about projects that use morecantile's default tms set without allowing overrides.
I'm not sure to understand this
Morecantile is a tool that lets you define and use TMS grid, having a set of defaults in the project is a bonus but isn't the goal of the project.
Yes it's possible to make PRs against them to accept a tms set as a parameter but my suggestion could be a very cool mechanism to support
I'm open for suggestion but right now I have no idea how it could look 🤷
There was a problem hiding this comment.
err "like" it but not exactly it, also not working currently
what is not working?
…think this is sufficient for the moment
|
Different file in the repository (planetcantile/defaults.py) , no longer broken due to recent pushes I made to fix- Dr. Andrew AnnexOn Dec 13, 2022, at 2:07 PM, Vincent Sarago ***@***.***> wrote:
@vincentsarago commented on this pull request.
In planetcantile/data/generate.py:
for tmsp in crss:
# create the tms object
tms = morecantile.TileMatrixSet.custom(**asdict(tmsp))
_d = json.loads(tms.json(exclude_none=True)) # get to pretty printed json
- with open(f'./{tmsp.crs.name}_tms.json', 'w') as dst:
+ with open(f'./{tmsp.identifier}_tms.json', 'w') as dst:
# dump to json
json.dump(_d, dst, indent=4, ensure_ascii=False)
print(f'wrote {dst.name}')
err "like" it but not exactly it, also not working currently
what is not working?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
I am going to try to merge this PR and clean up things a bit to ensure that the work is captured. I've saved a PDF of the page saving all of the discussions above just in case but will leave the pr up to allow the conversation to continue a bit here until it needs to move on to other prs/issues |
Went with urllib to keep everything in the standard library for pulling the input WKT2.