Error while adding a Project via the API

[Dryocopus]

Not sure what’s going on here.

When trying to add a Project like this:

[
{'uri': 'os-prj:VTCA', 'name': 'Virtual Tomato CropA', 'shortname': 'VTCA', 
'start_date': '2010-02-20', 'end_date': '2015-02-20', 'description': 'VTC aims at building a Digital Twin for the Tomato Crop', 
'objective': 'Building a Digital Twin for the Tomato', 'financial_funding': 'NWO'}
]

I get an error like this:

(500) Reason: HTTP response headers: HTTPHeaderDict({'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'origin, content-type, accept, authorization', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, OPTIONS, HEAD', 'Vary': 'accept-encoding,accept-encoding', 'Content-Type': 'application/json', 'Content-Length': '5226', 'Date': 'Mon, 05 Jul 2021 12:35:14 GMT', 'Connection': 'close'}) HTTP response body: { "metadata" : { "pagination" : { "pageSize" : 0, "currentPage" : 0, "totalCount" : 0, "totalPages" : 0 }, "status" : [ ], "datafiles" : [ ] }, "result" : { "title" : "Unexpected internal error - java.lang.NullPointerException", "message" : null, "stack" : [ "org.opensilex.core.project.api.ProjectAPI.createProject(ProjectAPI.java:107)" ], "fullstack" : [ "org.opensilex.core.project.api.ProjectAPI.createProject(ProjectAPI.java:107)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base/java.lang.reflect.Method.invoke(Method.java:566)", "org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)", "org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)", "org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)", "org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)", "org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)", "org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)", "org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)", "org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)", "org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)", "org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)", "org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)", "org.glassfish.jersey.internal.Errors.process(Errors.java:292)", "org.glassfish.jersey.internal.Errors.process(Errors.java:274)", "org.glassfish.jersey.internal.Errors.process(Errors.java:244)", "org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)", "org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)", "org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)", "org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)", "org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)", "org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)", "org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)", "org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)", "org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)", "org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)", "org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)", "org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)", "org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)", "org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)", "org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)", "org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)", "org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:206)", "org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:295)", "org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)", "org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)", "org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)", "org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:552)", "org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)", "org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)", "org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)", "org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)", "org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)", "org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)", "org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)", "org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)", "org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)", "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)", "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)", "org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)", "java.base/java.lang.Thread.run(Thread.java:829)" ] } }

This time there is a manually generated uri. Some fields are not filled at all, but I tried all kinds of things and nothing helps.

The strange thing is: if I try to add (more or less) the same project via the web-interface, it keeps hanging in this state:

I have to say that both our PHIS installation and my download of the openClientTools might not be up to date. But this seems to be in the server logic.

[Isabelle-inrae]

Hi Peter,

I have several questions regarding this issue:

• 1st question:

When you say that the project "keeps hanging in this state", do you mean the "In progress" state? Would you like it to be in the "Not yet started" state?
That's a good point, at the moment we only have the "In progress" / "Finished" states, that's why your project is in the "In progress" state.

[Isabelle-inrae]

• 2nd question:
  I tried to create the project with your model on our test instance with the swagger and it was ok :

Could you try to create directly with the swagger by using the POST/core/Project with this file:
{"uri": "os-prj:VTCA", "name": "Virtual Tomato CropA", "shortname": "VTCA",
"start_date": "2010-02-20", "end_date": "2015-02-20", "description": "VTC aims at building a Digital Twin for the Tomato Crop",
"objective": "Building a Digital Twin for the Tomato", "financial_funding": "NWO"}

I would like to know whether the problem comes from the file, the swagger or the client.

[Isabelle-inrae]

Another possibility is that the swagger does not handle empty tables.
We will add a patch, but in the meantime can you try sending this file and tell us what you get?

{
'uri': 'os-prj:VTCA',
'name': 'Virtual Tomato CropA',
'shortname': 'VTCA',
'start_date': '2010-02-20',
'end_date': '2015-02-20',
'description': 'VTC aims at building a Digital Twin for the Tomato Crop',
'objective': 'Building a Digital Twin for the Tomato',
'financial_funding': 'NWO',
'website': null,
'related_projects': [],
'coordinators': [],
'scientific_contacts': [],
'administrative_contacts': [],
'experiments': []
}

[Dryocopus]

Hi Isabelle,

Thanks for your reply so far.

FYI: After some further investigations I think I now have a quite clear picture about where the errors come from (there are multiple different issues), but in order to prevent confusing communication, I'll dig to the bottom of it, provide a detailed list of scenarios and their effects. I'll also try to list possible solutions. But due to urgent other work that might take until the end of today or tomorrow.

[Dryocopus]

About your 1st question: that was just confusion from my side. I didn't realize that icons were used for the project state, and because indeed In progress didn't seem to match with the start and end date, I thought this was probably the state of saving the entry in the database.

Not yet started would indeed be clearer, but that's a minor issue now that I understand what this state means.

[Dryocopus]

The 2nd issue is much more fundamental. You seem to work around the opensilexClientToolsPython library, which is why you see different things. But the nature of the problem is that I cannot do that from my code...

By systematically trying a whole range of scenarios, I tried to get clear what is going on. See the attached Word document. It's good to be critical, because in the confusing range of things I tried, I might well have made some mistakes, but what it looks like is this:

1. The JSON should be with double-quotes, but that is business-as-usual, just some prints from the code in the initial issue show single quotes because that is how Python prints dictionaries and objects. Both the input files and the calls to the webservice use double quotes (See Postman: Uri filled. Array-fields empty. JSON with single quotes > EXCEPTION)
2. Leaving out the uri works with Postman. So with the plain API it is possible to let PHIS generate uris itself (See Postman: No uri. No array-fields. Double quotes in JSON > SUCCESS)
3. Leaving out the uri when working via Python scripts does not work, because the constructor of ProjectCreationDTO still adds the uri-field and fills it with None. That causes a Nullpointerexception on the Web service (See Swagger: No uri. No array-fields. Double quotes in JSON > EXCEPTION).
4. The instance of post_creation_dto that is returned by the constructor of ProjectCreationDTO can be postprocessed in order to remove the unwanted uri attribute (and others as well). However, that still goes wrong on the server. Not sure why, probably there is still some intermediate processing on the way to the server (See Swagger: No uri (with postprocessing). No array-fields. Double quotes in JSON > EXCEPTION)
5. Adding the uri but setting it to an empty string or None doesn’t fix the issue (See Swagger. Uri as empty string. No array-fields. Double quotes in JSON)
6. Adding an uri filled with a value doesn’t work either (see Swagger. Uri filled. No array-fields. Double quotes in JSON > EXCEPTION)
7. Array fields are not required (See Postman: No uri. No array-fields. Double quotes in JSON > SUCCESS)

Summarizing:

• There is a generic problem because if you leave the uri out, the opensilexClientToolsPython generate an empty field for it. This isn’t accepted by the service logic. So it isn’t possible to let PHIS autogenerate uris if you work via the API-tools. This is the same is for Issue Presumed bug in OrganisationsAPI.create_infrastructure_facility #2 in Github.
• More fundamentally, for Projects it doesn’t even seem possible to load an entity with a pre-filled uri. That means that there is simply no way to add a project via the API.
• Obviously, new projects are rare, so in theory they can be added manually via the web interface, but that might interfere with some workflows. And the whole thing raises questions about stability.

Regarding possible solutions:

1. I currently don't see any way to get around the issue from my code.
2. Ideally the opensilexClientToolsPython should be improved not to generate an uri-field automatically. But building this library manually doesn't seem feasible, and I currently don't know if it is possible to tune the Swagger code-generation in such a way that the issue can be solved that way
3. So the only option I currently see is to modify the PHIS webservice in order to tolerate empty uri-fields and ignore them.

The fact that even prefilled uri's don't work here is an issue in itself. I have to think if I can do anything to get more clarity on this, but it has probably to be debugged by the webservice-team.

Attachment:
PHIS_Issue_opensilexClientToolsPython_Issue5_v0.1.docx

[JeanEudesH]

Hi Peter, Thank you very much for the investigation. I struggle a bit to reproduce the errors. I'm testing in the demo instance that is in a later version. There seem to be a lot less issues and exceptions are not happening.
You can find in attachment the file I use to test. I hope it helps.
If you can update your instance to the later version I hope it will solve these problems. That are indeed coming from the web services and not really the package.

`# TEST debug
import opensilexClientTools.api

pythonClient = opensilexClientTools.ApiClient()
pythonClient.connect_to_opensilex_ws( username="XXXXXXXXXX", password="XXXXXXXX", host="http://138.102.159.36:8084/rest")
project_api = opensilexClientTools.ProjectsApi(pythonClient)
project_DTO = opensilexClientTools.ProjectCreationDTO(uri ='os-proj:vtcBA',name="Virtual Tomato CropBA", shortname="VTCA3", start_date="2010-02-20", end_date="2015-02-20", description="VTC aims at building a Digital Twin for the Tomato Crop", objective="Building a Digital Twin for the Tomato", financial_funding="NWO", website='')
project_api.create_project(body=project_DTO)
`

[Isabelle-inrae]

To complete what Jean-Eudes said, we have just published a new release. I haven't had time to communicate on it yet, but you can find it here : https://github.com/OpenSILEX/opensilex/releases/tag/1.0.0-beta%2B2

You can update your instance with it.
Please note that for this version there is a change in the Data model: a data can only be associated to one scientific object and not to a list.
If you have inserted data or data files in your instance, here update scripts to play at the time of the release deployment :
- Scripts for data :
db.getCollection('data').aggregate([{$addFields: {"scientificObject": {$arrayElemAt: [ "$scientificObjects", 0 ]}}}, { $out : "newData" }])
db.getCollection('data').drop()
db.getCollection('newData').updateMany({}, {$unset: {"scientificObjects": 1}})
db.getCollection('newData').updateMany({"scientificObject":null}, {$unset: {"scientificObject": 1}})
db.getCollection('newData').reIndex()

- Scripts for datafiles :
db.getCollection('file').aggregate([{$addFields: {"scientificObject": {$arrayElemAt: [ "$scientificObjects", 0 ]}}}, { $out : "newfile" }])
db.getCollection('file').drop()
db.getCollection('newfile').updateMany({}, {$unset: {"scientificObjects": 1}})
db.getCollection('newfile').updateMany({"scientificObject":null}, {$unset: {"scientificObject": 1}})
db.getCollection('newfile').reIndex()
db.getCollection('newfile').renameCollection("file")

[Dryocopus]

Hi Jean-Eudes,

Are you suggesting updating the clientTools or the PHIS-installation on the server?

The clientTools seem to be the latest tagged version (opensilexClientToolsPython 2.0.0b0), so most likely that will not help.

I'm not sure what version our PHIS-installation is, but I believe it's the beta release from April. Sven is currently handling these things, and because his time is limited it may take some time to update that version if there is a new one.
Just got the additional comment from Isabelle, we will try to upgrade.
I don't think we need update scripts, all stuff in our installation is just for experimenting.

But in the meantime, I'm a bit puzzled by your code because you refer to 'import opensilexClientTools.api'. I use opensilexClientToolsPython. Is that different? There are various clientTools in your repository and I cannot find one called opensilexClientTools. If i just rename the library in your code to opensilexClientToolsPython, I get an error:
connect_to_opensilex_ws() got an unexpected keyword argument 'username'
So that's probably not what I should do.

But is it an option that I redirect my own logic to your demo-installation and try it there? I already have a username password for it. I can cleanup what I add afterwards. I'm not planning any changes besides adding a Project and perhaps Facility.

By the way, I've now tried to upload a lot of entity-types (most of them in a very basic way, without much associated entities) and most of that went ok. So the Project (Issue #5) and Facility (Issue #2) seem to be the only ones left that gives some problem. Next week, I'll try some more realistic scenarios.
(After out session from the end of May, I had to do a lot of work in other areas, hence the delay in the PHIS-area).

[JeanEudesH]

Are you suggesting updating the clientTools or the PHIS-installation on the server?

I was reffering updating the PHIS install with web services update. The web services handle these errors, as far as I see.

I use opensilexClientToolsPython. Is that different?

It's me sorry, I didn't use the correct spelling. Code stands up to date below. I'm sorry

But is it an option that I redirect my own logic to your demo-installation and try it there?

This is the option I use, to make sure my test don't pollute any database. I can erase afterward. So maybe when you try figure out, what is the correct approach, you can also use this one. And finally when the solution suits you, use it on your local installation.

`# TEST debug
import opensilexClientToolsPython

pythonClient = opensilexClientToolsPython.ApiClient()
pythonClient.connect_to_opensilex_ws( identifier="XXXXXX", password="XXXXX", host="http://138.102.159.36:8084/rest")
project_api = opensilexClientToolsPython.ProjectsApi(pythonClient)
project_DTO = opensilexClientToolsPython.ProjectCreationDTO(uri ='os-proj:vtcBAC',name="Virtual Tomato CropBA", shortname="VTCA3", start_date="2010-02-20", end_date="2015-02-20", description="VTC aims at building a Digital Twin for the Tomato Crop", objective="Building a Digital Twin for the Tomato", financial_funding="NWO", website='')
project_api.create_project(body=project_DTO)
`

[Dryocopus]

Ok, I'll try that, then I can cross-test various combinations of logic and installations and find out what caused the issue.
Thanks.

[Isabelle-inrae]

Peter,
in addition to what you've been told, we will be generating a new version of the R/Python client (probably early next week) that will be compatible with this new 1.0.0-beta+2 release
We'll let you know when this new package is available (you can find it with the others here: https://github.com/OpenSILEX/opensilexClientToolsPython/releases)
Have a nice weekend
Isabelle

[Dryocopus]

Sorry, it appears that this was a bug at my side. Effectively, I sent a request without a body, which obviously made it impossible to store anything.

Only thing at the PHIS-side is that it would be good to have a better error-handling. For that reason, I'll not yet close the issue myself, I'll leave it to the PHIS-team to look if anything can be improved and close the issue if you consider that appropriate.

Regarding the new release as announced by Isabelle, I didn't yet try that, because I have to wait until Sven has time to install it. So it is possible that this version already fixes the issue, but I cannot yet test that.

Up to this point, thanks for your help.