Dynamic data-driven layers

[grtjn]

We are working on a project that ingests sets of geojson files, with the idea that each of these files will represent a single layer. It concerns mostly static objects like roads, points of interests etc. There might be many such files, and it would be cumbersome to manage definitions for all of these. Wondering if a dynamic, data-driven approach would be achievable..

[tdiepenbrock]

We implemented a project with 620+ services based on OWL ontology classes for one of our clients. The keys are 1) rigorous object type definitions (in this client’s case it was via OWL), 2) auto-generation of TDE templates based on those object type definitions, and 3) auto-generation of service descriptors, again based on the object type definitions. I imagine we could do this via entity type definitions as well.

[grtjn]

In our case things are much simpler. We have a generic GeoFeatures es object that we use for all features, so we can suffice with one model, and one tde. It is really the service descriptors that is the complex part in our case. You cannot generate those out of the model, it is data-driven. It would be possible to generate a descriptor with each upload of a set, but calculated at runtime would be easier. Just not sure if that would be feasible or not. Wondering if one could provide an extra TDE just to generate layer specifics for instance..

[jkerr5]

Can you provide an example of what you would use as input and what you would want the descriptor to end up as? They could be generated on the fly but that would amount to creating some sort of descriptor descriptor :-)

[grtjn]

It is a bit difficult to wrap your head around this, without having some to look at, so let me provide some code. You can probably guess for which project this would be:

let $layers :=
  for $layer in distinct-values(collection("GeoFeature")//featureCollectionId)
  for $type in distinct-values(cts:search(collection("GeoFeature"), cts:json-property-value-query("featureCollectionId", $layer))//geometry/type)
  (:let $sample := subsequence(cts:search(collection("GeoFeature"), cts:json-property-value-query("featureCollectionId", $layer)), 1, 1):)
  order by $layer, $type
  return object-node {
    "name": $layer || " - " || $type,
    "description": "All " || $type || " geometries for layer " || $layer,
    "geometryType": $type,
      "idField": "OBJECTID",
      "readOnly": true(),
      "minScale": 25000,
      "maxScale": 0,
      "displayField": "location",
      "extent" : object-node {
        "xmin" : -7000,
        "ymin" : 289000,
        "xmax" : 300000,
        "ymax" : 629000,
        "spatialReference" : object-node {
          "wkid" : 4326,
          "latestWkid" : 4326
        }
      },
      "geometry" : object-node {
        "geometryType": $type,
        "format" : "geojson",
        "coordinateSystem" : "wgs84",
        "xpath" : "/envelope/instance/GeoFeature/geometry"
      },
      "boundingQuery" : object-node {
        "jsonPropertyScopeQuery": object-node{
          "property": array-node {
            "geometry"
          },
          "query": object-node{
            "jsonPropertyValueQuery": object-node{
              "property": array-node {
                "type"
              },
              "value": array-node {
                $type
              },
              "options": array-node {
                "case-sensitive",
                "diacritic-sensitive",
                "punctuation-sensitive",
                "whitespace-sensitive",
                "unstemmed",
                "unwildcarded",
                "lang=en"
              }
            }
          }
        }
      },
      "schema" : "SC",
      "view" : "GeoFeatures"
  }
for $layer at $index in $layers
let $json := xdmp:from-json($layer)
let $_ := map:put($json, "layer", $index)
return xdmp:to-json($json)

Some details might need more thought, like extent and such, but the most important element is featureCollectionId. That is the element whose value basically references the layer. I added some safeguards in case there is a mix of feature types in that feature collection (each feature collection represents a single geojson bulk file, so it is fairly safe to assume it will be single type, but just to be sure).

The geometry is stored in GeoFeature/geometry (always that same path for many of the layers), so it is also fair to assume we can come up with one TDE to provide the props. Although we have a challenge there as mentioned in #69.

It would be possible to generate descriptors and TDE's at deploy time, but geojson bulk files are loaded after deployment, and part of that will happen in automatic fashion. There are ways to employ triggers and such to do this at post-commit time, but having a way to dynamically retrieve those descriptors would make more sense.

In retro-spect not entirely sure it could be mapped on a TDE, but pretty sure you can get close. Either way, I could imagine you just specify the service or services, and inside you reference code or TDE to provide the layer descriptors..

[mfgumban]

One possibility would be a service descriptor "factory":

• It would be declared as a JSON document same as a service descriptor; could even be under the same required collection, it would just have a different JSON structure.
• It would contain a path referencing a XQuery/SJS module function
• The function accepts a string (feature service name) as input and returns sequence of JSON nodes that represent service descriptors. Could return an empty sequence.
• GDS already caches service descriptors in server variables, so invoking the function would become part of that process. Given a request, if it couldn't find any matching service descriptor document, it will invoke the factory methods one by one until one of them returns a matching descriptor, then add it to the cache.