myrec$id empty

[oggioniale]

Dear eblondel,
I'm testing zen4R for upload some research products for LTER Research Infrastructure the final idea is to propose a ShinyApp.
At this moment I have a problem for upload a records both in the sandobox and in a production one.
In the first cases, when I set the ZenodoManager with url = "http://sandbox.zenodo.org/api"
I receive the answer:
[zen4R][ERROR] ZenodoManager - Error while depositing record: The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.
In the second case, try to publish the record directly to Zenodo production, the record is deposited [zen4R][INFO] ZenodoManager - Successful record deposition but when I try to publish the record I don't understand how to retrive the id of the record deposited.

My script for the second case below

# devtools::install_github("eblondel/zen4R")
zenodo <- zen4R::ZenodoManager$new(
  # url = "http://sandbox.zenodo.org/api", # to test record management before going with the production Zenodo e-infrastructure
  token = tokenLTERItaly,
  logger = "DEBUG" # "INFO"
)

###
# record management
###
siteName <- "Lago Como"
deimsID <- "https://deims.org/58036d71-8141-40c3-a0f2-50b8bd1bcddc"
# Create an empty record
myrec <- zen4R::ZenodoRecord$new()

# Fill a record
myrec$setUploadType("image")
myrec$setImageType("figure")
myrec$setTitle(paste0("LTER-Italy ", siteName, " site figure"))
myrec$setDescription(paste0("Figure of the LTER-Italy ", siteName, " research site DEIMS-ID ", deimsID))
myrec$addCreator(firstname = "Alessandro", lastname = "Oggioni", affiliation = "CNR-IREA", orcid = "0000-0002-7997-219X")
# zenodo$getLicenses() # to fetch the list of licenses
myrec$setLicense("CC-BY-SA-4.0")
myrec$setAccessRight("open")
myrec$setVersion("1.0")
myrec$setLanguage("eng")
myrec$setKeywords(c(siteName, deimsID, "LTER-Italy", "map of site"))
# myrec$addRelatedIdentifier("compiles", "") add the DOI of eLTER-SiteFigure functions
# zenodo$getCommunityById("lter-italy")
myrec$setCommunities("lter-italy")
# myrec$setGrants("654359")

# Deposit/update a record
mrec <- zenodo$depositRecord(myrec)

# Publish a record
zenodo$publishRecord(myrec$id)

# Upload file
zenodo$uploadFile("path/to/your/file", myrec$id)

How I do it? Thanks a Lot
oggioniale

[eblondel]

Hello @oggioniale

• For this message [zen4R][ERROR] ZenodoManager - Error while depositing record: The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required:

The token isn't the same in sandbox and production. In each instance of Zenodo, you have to pick-up a token from your Zenodo account, checking deposit:actions and deposit:write.

• For the issue with using the code:

By creating a blank record with ZenodoRecord$new(), this is a local object from zen4R, hence the id is initially set NULL. Once you deposit your record, you can get the same record but fetched from Zenodo (deposited), and there you will get an id (and more for example you will inherit a reserved DOI for this record). In your code, you just have a typo and return the result of depositRecord as mrec object. Replace it with:

myrec <- zenodo$depositRecord(myrec)

In that case this will send your local record to Zenodo, deposit it, and get back the fetched record (with id, doi, etc) into the same R object. You can then publish the record picking up myrec$id as you did

An alternative to create a blank local record to start drafting it is to create directly a blank record in Zenodo, and then you can edit it, and trigger deposit. In that case the record, since deposited early in Zenodo, will have id, prereserved doi etc...

myrec <- zenodo$createEmptyRecord()

Apart from that I see that you are from LTER-Italy, it may be interesting that you interact with the french node, related to your work on pushing metadata / manage spatial data (i've seen we exchanged in the past on ows4R) etc I'm in contact with people from "Zone Ateliers RZA" (https://cat.opidor.fr/index.php/E-LTER-France_RZA) pilar of LTER-France. Some data managers from there already use zen4R or other related packages. On my side i'm developing and puts together zen4R, ows4R, etc. This package is named geoflow, which aims is to automate/orchestrate (meta)data publication targeting different systems (eg. Geonetwork/Geoserver for spatial (meta)data, Zenodo, Dataverse, etc). RZA is envisaging to use geoflow to carry out their (meta)data management tasks. This could be of interest of you as well, the only point is that because i developed a lot on geoflow recently, i'm late on documentation, so you will have to be patient to get a complete user manual... But in case you are interested feel free to tell me.

Best

[oggioniale]

Hi @eblondel,
I didn't realize there was a typo. Sorry for that!
After that I uploaded my first image in Zenodo of LTER-Italy community.

I have used in the past ows4R for some project. The geoflow package is very cool. Is it a package for manage data and curates this in data management sofware that have APIs? In eLTER-RI I'm involved in two H2020 projects for improve the tools for data management and within the CNR-IREA institute we develop a sofware based on GeoNode named GET-IT (the github project) that we propose, in eLTER community, as a country data node.

DEIMS-SDR is a repository of the metadata of sites and data, GeoNetwork is a center catalogue of the metadata dataset, B2Share is used for provide DOI to dataset, and finally GET-IT is for the countries that are autonomus for share data.
How do you think is possible to connect these?

[eblondel]

Hi @oggioniale geoflow is aimed to be a kind of toolbox for people that want to automate data processings and publication flows targeting one ore software, using APIs, and when available standard protocols. So far i've pushed publication API actions for packages I'm the author, as it eases their integration:

• data for Geoserver with geosapi although it still needs to add support to raster / coverage, that i'd like to integrate together with the developement of an WCS Client ows4R
• ISO/OGC metadata including ISO 19115, 19119 and 19110 as iSO 19139 XML creation (with geometa
• Publication of XML ISO 19319 metadata into Geonetwork with geonapi and any CSW catalogue with ows4R. On that i've started investigating on we could interact with Geonode, but this in stand-by (see Publish metadata on geonode by using CSW-T ows4R#25) @kikislater and @juldebar made some tests on this, but AFAIK they are specific configs in Geonode CSW that prevent from connecting and pushing metadata.
• Publication of Zenodo records with present package

I'm currently looking how a similar publication action could be set-up for Dataverse, which many research institutions used as catalogue, and there is a plan to develop 2 actions related to biodiversity (meta)data:

• action to generate EML metadata
• action to push EML metadata through API such as Metacat

Recently i've developed and released another package on CRAN to deal with the Open Collaboration Service (OCS) protocol: ocs4R and set-up 2 basic wrappers for NextCloud and OwnCloud.
I'm not 100% sure, but i think that B2share (and/or B2drop) rely on one of these OCS-compliant cloud plateforms. I had started looking at it. One idea i have in mind would be to set-up an action to interact with these kind of software, with same spirit, deposit files/metadata and assign DOI when possible.

Geoflow is metadata driven. In other words we defined a pivot metadata model, initially inspired from Dublin Core, and expandable to cover more exhaustive metadata standards (such as ISO/OGC or EML). The idea is that people can rationalize their intervention on a single metadata entry point, and trigger multiple actions to target diverse systems (eg Geonetwork/Geoserver for spatial, EML for biodiversity oriented catalogues, general Zenodo/Dataverse for DOI publication). Depending on the actions triggered, these can be inter-related, e.g. you publish some DOI, this DOI can be reflected properly in the ISO/OGC metadata. Vice-versa, you can push different metadata sheets as resources into a Zenodo deposit together with data.

The pivot metadata model is defined with an R class model in geoflow, and we provided wrappers to fit profile (user, R developer). For that purpose we set-up a simplified tabular data structure as entry point that could be either CSV, XLSX, Google spreadsheet, or a table in a DBMS. Three kinds of metadata entry points can be set:

• entities (where 1 table row is one dataset metadata),
• contacts (serves as entry point for getting the persons/institutes that are attached to dataset metadata )
  *dictionary (serves as data dictionary for advanced cases, eg when we want to produce ISO 19110 feature catalogues to describe data structures)
  On the entry points, they are also perspectives. Tabular data entry is targeting users that start from scratch, but other people expressed the wish to start from different entry point, eg another CSW catalogue for (meta)data, LDAP for contacts, Thredds for raster data etc

Well, I don't want to very verbose. Geoflow is going to expand further in the near future, with some developments for which I have funding opportunities yet, and other not yet (raster support). You can have a look to geoflow, I believe this could be of interest for you, and anyway we can keep in touch.

Last but not least, one of my goal is to foster production/publication of metadata not as ultimate goal, but also for to exploit metadata as driver for advanced data exploration. In this purpose i've been developing the OpenFairViewer https://github.com/eblondel/OpenFairViewer, which focus on spatial data, and aims to offer a fully standard ISO/OGC metadata driven map viewer. It relies entirely on OGC CSW for browsing datasets, OGC WxS protocols for data access, and query is enabled by means of ISO 19115 dataset metadata and ISO 19110 structural metadata. This portal is being promoted in some H2020 project as well (which includes a collaboration with Italian CNR-ISTI), and in France some colleagues within the LTER-France network are considering a possible adoption. Its development is tightly bound to the geoflow one with the idea that what you enable in geoflow as spatial data/metadata, is exploitable within OpenFairViewer.

[PietrH]

I'm having a similar problem where my new record newrec returns the message [zen4R][INFO] ZenodoManager - Successful record deposition when I deposit, but I still get NULL for newrec$id

My code (left out some lines unrelated to zenodo):

zenodo <- ZenodoManager$new(
  url = "http://sandbox.zenodo.org/api",
  token = "...", 
  logger = "INFO"
)

projectID<-"4035235"



# creating a new zenodo deposition ----------------------------------------

newrec <- ZenodoRecord$new()
# newrec <- zenodo$createEmptyRecord()

# fill Zenodo metadata ----------------------------------------------------


newrec$setTitle(paste("DoeDat Camera Trap Project",projectID,"(Meise Botanic Garden)"))

newrec$setDescription(
  paste0(
    'A project on the <a href=\"https://www.doedat.be\">DoeDat platform</a>',
    ' of <a href=\"https://www.plantentuinmeise.be\">Meise Botanic Garden</',
    'a>. Footage of camera traps is offered to volunteers for identificatio',
    'n of the animals visible, if any. This deposition is an archive of the',
    'images in this project as well as the transcription results formatted ',
    'in Darwin Core.'
    )
  )

newrec$setUploadType("image")
newrec$setImageType("photo")
newrec$addCreator(name="Meise Botanic Garden")
newrec$addRelatedIdentifier("isAlternateIdentifier","http://apm-ipt.br.fgov.be:8080/ipt/")
newrec$setKeywords(c('Biodiversity','Wildlife','Terrestrial','DoeDat','Citizen Science'))
newrec$setLicense("CC0-1.0")
newrec$setImprintPublisher("Meise Botanic Garden")
newrec$setCommunities(c("mbg-cameratrap",
                        "meisebotanicgarden")) # possibly doesn't exist on sandbox
newrec$addContributor("Quentin","Groom","ContactPerson","Meise Botanic Garden","0000-0002-0596-5376")
newrec$addSubject(term="Meise Botanic Garden",
                  identifier="http://purl.org/dc/terms/rightsHolder")
newrec$addSubject(term="BR",
                  identifier="http://rs.tdwg.org/dwc/terms/institutionCode")
newrec$addSubject(term="Camera trap images Meise Botanic Garden",
                  identifier="http://rs.tdwg.org/dwc/terms/datasetName")
newrec$addSubject(term="Woodland",
                  identifier="http://rs.tdwg.org/dwc/terms/habitat")
newrec$addSubject(term="Machine Observation",
                  identifier="http://rs.tdwg.org/dwc/terms/basisOfRecord")
newrec$addSubject(term="https://creativecommons.org/publicdomain/zero/1.0/legalcode",
                  identifier="http://purl.org/dc/terms/license")
newrec$addSubject(term="BE",
                  identifier="http://rs.tdwg.org/dwc/terms/countryCode")



# deposit Zenodo record ---------------------------------------------------

newrec <- zenodo$depositRecord(newrec)

Thank you for your help!

[eblondel]

Hi @PietrH when using the above code on sandbox, I get this:

[zen4R][ERROR] ZenodoManager - Error while depositing record: Validation error. 
[zen4R][ERROR] ZenodoManager - Error: metadata.communities - Invalid communities: meisebotanicgarden

If i remove the line to set the communities, the R code works, I get an output and newrec$id is set.

[PietrH]

Hello @eblondel , thanks for looking into this.

When I leave my code as is, I get:

> newrec <- zenodo$depositRecord(newrec)
[zen4R][INFO] ZenodoManager - Successful record deposition

With

> newrec$id
NULL

Which is a possible bug as it reports successful deposition, but doesn't actually deposit?

Then after removing the line that adds the communities, I still have the same results. Here is my full terminal printout:

> newrec <- ZenodoRecord$new()
> newrec$setTitle(
+   paste("DoeDat Camera Trap Project", projectID, "(Meise Botanic Garden)"))
> newrec$setDescription(
+   paste0(
+     'A project on the <a href=\"https://www.doedat.be\">DoeDat platform</a>',
+     ' of <a href=\"https://www.plantentuinmeise.be\">Meise Botanic Garden</',
+     "a>. Footage of camera traps is offered to volunteers for identificatio",
+     "n of the animals visible, if any. This deposition is an archive of the",
+     "images in this project as well as the transcription results formatted ",
+     "in Darwin Core."
+   )
+ )
> newrec$setUploadType("image")
> newrec$setImageType("photo")
> newrec$addCreator(name = "Meise Botanic Garden")
> newrec$addRelatedIdentifier("isAlternateIdentifier",
+                             "http://apm-ipt.br.fgov.be:8080/ipt/")
[1] TRUE
> newrec$setKeywords(c(
+   "Biodiversity",
+   "Wildlife",
+   "Terrestrial",
+   "DoeDat",
+   "Citizen Science"
+ ))
> newrec$setLicense("CC0-1.0")
> newrec$setImprintPublisher("Meise Botanic Garden")
> # newrec$setCommunities(c(
>   # "mbg-cameratrap"
>   # ,"meisebotanicgarden"
> # )) # possibly doesn't exist on sandbox
> newrec$addContributor("Quentin",
+                       "Groom",
+                       "ContactPerson",
+                       "Meise Botanic Garden",
+                       "0000-0002-0596-5376")
> newrec$addSubject(
+   term = "Meise Botanic Garden",
+   identifier = "http://purl.org/dc/terms/rightsHolder"
+ )
> newrec$addSubject(
+   term = "BR",
+   identifier = "http://rs.tdwg.org/dwc/terms/institutionCode"
+ )
> newrec$addSubject(
+   term = "Camera trap images Meise Botanic Garden",
+   identifier = "http://rs.tdwg.org/dwc/terms/datasetName"
+ )
> newrec$addSubject(
+   term = "Woodland",
+   identifier = "http://rs.tdwg.org/dwc/terms/habitat"
+ )
> newrec$addSubject(
+   term = "Machine Observation",
+   identifier = "http://rs.tdwg.org/dwc/terms/basisOfRecord"
+ )
> newrec$addSubject(
+   term = "https://creativecommons.org/publicdomain/zero/1.0/legalcode",
+   identifier = "http://purl.org/dc/terms/license"
+ )
> newrec$addSubject(
+   term = "BE",
+   identifier = "http://rs.tdwg.org/dwc/terms/countryCode"
+ )
> newrec <- zenodo$depositRecord(newrec)

Even if I deposit with just a title and description, I get the same results. Is it possible there's something wrong with my key? If I run zenodo$getDepositionById(653087) I get:

[zen4R][INFO] ZenodoRequest - Fetching http://sandbox.zenodo.org/api/deposit/depositions?q=recid:653087&size=10&page=1&all_versions=1 
[zen4R][INFO] ZenodoManager - Successfully fetched list of depositions! 
[zen4R][WARN] ZenodoManager - No record for id '653087'! 
NULL

Trying to reach this test record https://sandbox.zenodo.org/record/653087, my key is set to actions and write permissions. I've used it before.

I'm on version Zen4R 0.4-3 running in R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out"