Json harvester #5942

fgravin · 2021-09-06T12:13:37Z

Continue the work started by @fxprunayre in #4034
Aligned with last main branch.

The goal is to be able to harvester Opendata catalog native API endpoints (CKAN, Opendatasoft, esri).
Loop on JSON datasets and map each object to a metadata record using a dedicated XSL transformation.

Exemple of configuration

OPENDATASOFT
URL https://metropole-europeenne-de-lille.opendatasoft.com/api/datasets/1.0/search
loopElement /datasets
recordIdPath datasetid
toISOConversion OPENDATASOFT-to-DCAT2
ESRI
URL https://data-atmo-hdf.opendata.arcgis.com/data.json
loopElement /dataset
numberOfRecordPath /result/count
recordIdPath identifier
pageFromParam start
pageSizeParam rows
toISOConversion ESRIDCAT-to-DCAT2

A simple harvester which takes a URL expecting for now a JSON document and loop over document identified by a JSONPointer and applying an XSL to convert to ISO format. This should allow GeoNetwork to harvest some of the opendata portal providing all various search API providing JSON response usually.

…w-sortby

fgravin · 2021-09-07T07:28:50Z

harvesters/src/main/java/org/fao/geonet/kernel/harvest/harvester/simpleUrl/Harvester.java

+
+                        nodes.forEach(record -> {
+                            Element xml = convertRecordToXml(record);
+                            uuids.put(record.get(params.recordIdPath).asText(), xml);


@fxprunayre, @josegar74 if the identifier is an URI (like in ESRI DCAT eg "identifier": "https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d" it set the URI as the uuid and it does not work when I open the metadata page.
I did a hack for my test:

uuids.put(record.get(params.recordIdPath).asText().split("/datasets/")[1], xml);

but I don't know how we could handle that properly, in any situation, would you have any suggestion ?
having a regexp in the harvester setting to extract the uuid but seems a bit tricky for the admins.
Thanks

@fxprunayre did this PR #5736 that probably helps, but needs to enable some specific configuration.

If the identifiers have this format: http(s)://URL/UUID, maybe an option is when converting the JSON to ISO19139, set the gmd:fileIdentifier to the UUID part of the identifier element in JSON and store the full identifier in gmd:identifier element. That should not require any hack in the UI code, but not sure if it's really "correct" from the metadata content point of view.

Thanks @josegar74, yes I think it's the way to go
I would just keep the uuid for the uuid and geep the uri for the resourceIdentifier.
But it means while harvesting, I have to know where is the uuid in the URI to extract it.

https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d will extract bac17d7d05a34242a8b22c535ecdb13d

This commit is a squash of geonetwork/core-geonetwork#5942 A simple harvester which takes a URL expecting for now a JSON document and loop over document identified by a JSONPointer and applying an XSL to convert to ISO format. This should allow GeoNetwork to harvest some of the opendata portal providing all various search API providing JSON response usually. Harvester / Simple URL / Paging and basic opendatasoft support. Json harvester: fix merge conflicts jsonHarvester: handle JSONLD format with @ in tag names jsonHarvester: add ESRI JSONLD DCAT transformation hack: to remove, extract uuid from URIs

This commit is a squash of geonetwork/core-geonetwork#5942 A simple harvester which takes a URL expecting for now a JSON document and loop over document identified by a JSONPointer and applying an XSL to convert to ISO format. This should allow GeoNetwork to harvest some of the opendata portal providing all various search API providing JSON response usually. Harvester / Simple URL / Paging and basic opendatasoft support. Json harvester: fix merge conflicts jsonHarvester: handle JSONLD format with @ in tag names jsonHarvester: add ESRI JSONLD DCAT transformation hack: to remove, extract uuid from URIs jsonHarvester: extract uuid from identifier https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d will extract bac17d7d05a34242a8b22c535ecdb13d

used by ODS to compute exports links

fgravin · 2021-09-20T13:16:19Z

@fxprunayre does it follow what you have initiated, do you approve ?
Can we move forward with this one please.
Thanks

jahow

Not really an expert on harvesters but I think this makes sense as a first iteration. Ideally this harvester should not be used directly as it is quite low level, and the user should be able to choose between ESRI DCAT, OpenDataSoft, CKAN etc.

fgravin · 2021-09-22T08:15:23Z

Thanks @jahow

This commit is a squash of geonetwork/core-geonetwork#5942 A simple harvester which takes a URL expecting for now a JSON document and loop over document identified by a JSONPointer and applying an XSL to convert to ISO format. This should allow GeoNetwork to harvest some of the opendata portal providing all various search API providing JSON response usually. Harvester / Simple URL / Paging and basic opendatasoft support. Json harvester: fix merge conflicts jsonHarvester: handle JSONLD format with @ in tag names jsonHarvester: add ESRI JSONLD DCAT transformation hack: to remove, extract uuid from URIs jsonHarvester: extract uuid from identifier https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d will extract bac17d7d05a34242a8b22c535ecdb13d

fxprunayre · 2023-05-12T04:45:47Z

schemas/iso19115-3.2018/src/main/plugin/iso19115-3.2018/convert/fromJsonLdEsri.xsl

+    <xsl:strip-space elements="*"/>
+
+    <xsl:template match="/record">
+      <xsl:variable name="cataloglang" select="'fr'"></xsl:variable>


Not sure french language hard coded value here is representative of the variety of ESRI users. Not sure everyone using this want to create metadata record in french. You can check how other harvester are handling the case of a source not providing language information (eg. OGCWxS)

This commit is a squash of geonetwork/core-geonetwork#5942 A simple harvester which takes a URL expecting for now a JSON document and loop over document identified by a JSONPointer and applying an XSL to convert to ISO format. This should allow GeoNetwork to harvest some of the opendata portal providing all various search API providing JSON response usually. Harvester / Simple URL / Paging and basic opendatasoft support. Json harvester: fix merge conflicts jsonHarvester: handle JSONLD format with @ in tag names jsonHarvester: add ESRI JSONLD DCAT transformation hack: to remove, extract uuid from URIs jsonHarvester: extract uuid from identifier https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d will extract bac17d7d05a34242a8b22c535ecdb13d

fxprunayre and others added 6 commits September 17, 2019 09:35

Harvester / Simple URL / Paging and basic opendatasoft support.

ed05387

Merge remote-tracking branch 'fxp/feature/harvester/json' into fix-cs…

cc66de5

…w-sortby

Json harvester: fix merge conflicts

f760ed1

jsonHarvester: handle JSONLD format with @ in tag names

a76271e

jsonHarvester: add ESRI JSONLD DCAT transformation

c42d4ca

fgravin commented Sep 7, 2021

View reviewed changes

jahow mentioned this pull request Sep 8, 2021

Add the DCAT2 schema plugin georchestra/geonetwork#183

Merged

2 tasks

jsonHarvester: extract uuid from identifier

332ff96

https://data-atmo-hdf.opendata.arcgis.com/datasets/bac17d7d05a34242a8b22c535ecdb13d will extract bac17d7d05a34242a8b22c535ecdb13d

fgravin force-pushed the json-harvester branch from 4fc5872 to 332ff96 Compare September 9, 2021 08:04

fgravin added 2 commits September 15, 2021 12:27

ESRI & ODS tranformation to DCAT2

878b57e

jsonHarvester: add api& nodeUrl in XML for transfo

6fdb175

used by ODS to compute exports links

fgravin marked this pull request as ready for review September 15, 2021 10:31

fgravin requested a review from fxprunayre September 15, 2021 10:33

jahow approved these changes Sep 21, 2021

View reviewed changes

fgravin merged commit de14f1c into geonetwork:main Sep 22, 2021

fgravin deleted the json-harvester branch September 22, 2021 08:15

fxprunayre reviewed May 12, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Json harvester #5942

Json harvester #5942

fgravin commented Sep 6, 2021 •

edited

Loading

fgravin Sep 7, 2021

josegar74 Sep 7, 2021 •

edited

Loading

fgravin Sep 7, 2021

fgravin commented Sep 20, 2021

jahow left a comment

fgravin commented Sep 22, 2021

fxprunayre May 12, 2023 •

edited

Loading

Json harvester #5942

Json harvester #5942

Conversation

fgravin commented Sep 6, 2021 • edited Loading

fgravin Sep 7, 2021

Choose a reason for hiding this comment

josegar74 Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

fgravin Sep 7, 2021

Choose a reason for hiding this comment

fgravin commented Sep 20, 2021

jahow left a comment

Choose a reason for hiding this comment

fgravin commented Sep 22, 2021

fxprunayre May 12, 2023 • edited Loading

Choose a reason for hiding this comment

fgravin commented Sep 6, 2021 •

edited

Loading

josegar74 Sep 7, 2021 •

edited

Loading

fxprunayre May 12, 2023 •

edited

Loading