Skip to content
This repository has been archived by the owner on Jul 14, 2024. It is now read-only.

Incompletely created organizations #150

Open
bobeal opened this issue Jan 8, 2016 · 15 comments
Open

Incompletely created organizations #150

bobeal opened this issue Jan 8, 2016 · 15 comments

Comments

@bobeal
Copy link
Contributor

bobeal commented Jan 8, 2016

Hi,

We encounter a problem when creating organizations :

  • reproducible on @OzBruno browsers (FF and Chrome) but not on mines (also FF and Chrome).
  • only in production and not in pre-production
  • when creating an organization either from the store or from Network page
  • also encountered by some Agrilocal users

Logs related to the organization testCreaOrga1755 :

oasis-portal.log:1012558:| DEBUG| 17:57:06.989 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - It doesn't exist in DC Doing a datacore.saveResource() with : org,DCOrganization{legal_name='testCreaOrga1755', sector_type='COMPANY', country_uri='http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR', lang='fr', org_type='null', tax_reg_num='testCreaOrga1755'}
oasis-portal.log-1012559-| DEBUG| 17:57:06.990 | org.oasis_eu.spring.datacore.impl.DatacoreClientImpl - Saving into Resource: URI String is https://data.ozwillo.com/dc/type/orgfr:Organisation_0
oasis-portal.log-1012560-| DEBUG| 17:57:07.010 | kernelLogging.logRequestTimings - POST Request to https://data.ozwillo.com/dc/type/orgfr:Organisation_0 took 9 msoasis-portal.log:1012561:| DEBUG| 17:57:07.596 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - Setting the new DC URI into the dcOrganization : http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/testCreaOrga1755

The creation seems to have been working fine, the organization has a DC id and it is then created successfully in the Kernel.

But when I look at it in the playground, I see that it has been partially created (country and other required fields have been « lost ») :

{
   "@id" : "http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/testCreaOrga1755",
   "o:version" : 0,
   "@type" : [
      "orgfr:Organisation_0",
      "org:Organization_0",
      "odisp:Displayable_0",
      "relt:Target_0",
      "adrpost:Postal_0",
      "orgpu:PublicOrg_0"
   ],
   "dc:created" : 2016-01-07T17:55:57.000+01:00,
   "dc:modified" : 2016-01-07T17:55:57.792+01:00,
   "dc:creator" : c06581b1-796a-4459-9ba3-23874c890c6a,
   "dc:contributor" : c06581b1-796a-4459-9ba3-23874c890c6a,
   "org:legalName" : [
      {
         "v" : testCreaOrga1755,
         "l" : fr
      }
   ],
   "org:status" : Normal Activity,
   "org:sector" : Private,
   "org:altName" : [
      {
         "v" : ,
         "l" : fr
      }
   ],
   "org:regNumber" : testCreaOrga1755
}

Nearly at the same time (yes, there is a 1 minute and some seconds gap between the two servers), I see this in the DC log (but really don't know if it is related) :

07-01-2016 17:55:59 WebApplicationExceptionMapper [WARN] javax.ws.rs.ClientErrorException: HTTP 409 Conflict
at org.oasis.datacore.rest.server.DatacoreApiImpl.internalPostDataInType(DatacoreApiImpl.java:163)
at org.oasis.datacore.rest.server.DatacoreApiImpl.internalPostAllDataInType(DatacoreApiImpl.java:194)
at org.oasis.datacore.rest.server.DatacoreApiImpl.postAllDataInType(DatacoreApiImpl.java:179)
@mdutoo
Copy link
Collaborator

mdutoo commented Jan 8, 2016

I doubt this is a Datacore problem, it's most probably rather on portal side.

A few ideas:

  • maybe you haven't seen this part of the portal logs just before that:

| DEBUG| 17:57:06.989 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - It doesn't exist in DC Doing a datacore.saveResource() with : org,DCOrganization{legal_name='testCreaOrga1755', sector_type='COMPANY', country_uri='http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR', lang='fr', org_type='null', tax_reg_num='testCreaOrga1755'}

=> indeed there should be a country, and an odisp:name should have been computed out of all that, but I can't see what "other fields" would have been lost ?

(I don't know at all if this 409 Conflict error is related or not. But indeed, not properly handling a 409 (by re-attempting POST/PUT) could trigger such a problem, IF the Resource org were to be written in two POST/PUT requests, which it should not, though maybe the previous portal code or even the react front is badly written and do 2 requests)

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 10, 2016

When trying to modify the same org with the same user:

java.lang.NullPointerException org.oasis_eu.portal.services.dc.organization.DCOrganizationService.setDCIdOrganization(DCOrganizationService.java:337) org.oasis_eu.portal.services.dc.organization.OrganizationService.update(OrganizationService.java:147) org.oasis_eu.portal.front.my.network.NetworkAJAXServices.updateDCOrganization(NetworkAJAXServices.java:146)

because (as a consequence of above) there is no country (to take an acronym of):
https://github.com/ozwillo/ozwillo-portal/blob/master/portal-parent/ozwillo-portal-front/src/main/java/org/oasis_eu/portal/services/dc/organization/DCOrganizationService.java#L337

@bobeal
Copy link
Contributor Author

bobeal commented Jan 11, 2016

I doubt this is a Datacore problem, it's most probably rather on portal side.

Maybe. I don't know (and that's why I did not label this issue as a bug).

maybe you haven't seen this part of the portal logs just before that:
| DEBUG| 17:57:06.989 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - It doesn't exist in DC Doing a datacore.saveResource() with : org,DCOrganization{legal_name='testCreaOrga1755', sector_type='COMPANY', country_uri='http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR', lang='fr', org_type='null', tax_reg_num='testCreaOrga1755'}

=> indeed there should be a country, and an odisp:name should have been computed out of all that, but I can't see what "other fields" would have been lost ?

Yes, I saw them. I have even put them in my first comment. This is a log I've added to see what was sent to the DC and you only see here the fields I added to the log (and country and odisp:name are not one of them).

Other lost fields are the fields required to create an organization (street name, NAF code, ...). I'll add them to the log trace.

it shows that org_type is null => have you checked whether it only happens in this case and not in the opposite ? it would mean that it is a bug related to the changes done when adding the "null org type" case.

Yes, org_type is also null for a correctly created organization.

if you can reproduce it, debug (this is prod, but using a conditional breakpoint in Eclipse it is workable) or log the result of toNewResource() just before creation on line 183:

Ok, I'll try that.

(I don't know at all if this 409 Conflict error is related or not. But indeed, not properly handling a 409 (by re-attempting POST/PUT) could trigger such a problem, IF the Resource org were to be written in two POST/PUT requests, which it should not, though maybe the previous portal code or even the react front is badly written and do 2 requests)

If it was the case, I would see the two calls in the portal logs. And I don't.

When trying to modify the same org with the same user:

java.lang.NullPointerException org.oasis_eu.portal.services.dc.organization.DCOrganizationService.setDCIdOrganization(DCOrganizationService.java:337) org.oasis_eu.portal.services.dc.organization.OrganizationService.update(OrganizationService.java:147) org.oasis_eu.portal.front.my.network.NetworkAJAXServices.updateDCOrganization(NetworkAJAXServices.java:146)

Yes, I noticed that. But, as you said, that's just a consequence of the creation problem.

@bobeal
Copy link
Contributor Author

bobeal commented Jan 11, 2016

@OzBruno got a similar problem this morning :

| DEBUG| 09:16:49.914 | org.oasis_eu.spring.datacore.impl.DatacoreClientImpl - Fetching limited Resources: URI String is https://data.ozwillo.com/dc/type/org:Organization_0?start=0&limit=1&org:regNumber=SCD11012015&adrpost:country=http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR
| DEBUG| 09:16:49.952 | kernelLogging.logRequestTimings - GET Request to https://data.ozwillo.com/dc/type/org:Organization_0?start=0&limit=1&org:regNumber=SCD11012015&adrpost:country=http://data.ozwill
o.com/dc/type/geocofr:Pays_0/FR took 38 ms
| DEBUG| 09:16:49.996 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - Fetched 0 resources in 82 ms
| DEBUG| 09:16:49.996 | org.oasis_eu.portal.services.dc.organization.DCOrganizationService - It doesn't exist in DC Doing a datacore.saveResource() with : org,DCOrganization{legal_name='Société Coopérative Drômoise', sector_type='COMPANY', country_uri='http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR', lang='fr', org_type='null', tax_reg_num='SCD11012015'}
| DEBUG| 09:16:49.996 | org.oasis_eu.spring.datacore.impl.DatacoreClientImpl - Saving into Resource: URI String is https://data.ozwillo.com/dc/type/orgfr:Organisation_0
| DEBUG| 09:16:50.001 | kernelLogging.logRequestTimings - POST Request to https://data.ozwillo.com/dc/type/orgfr:Organisation_0 took 4 ms
| INFO | 09:16:50.047 | kernelLogging.logFullErrorResponses - Full response body: {"code":"org.oasis.datacore.rest.server.resource.ResourceException","resource":{"@id":"http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/SCD11012015","o:version":null,"@type":["orgfr:Organisation_0","org:Organization_0","odisp:Displayable_0","relt:Target_0","adrpost:Postal_0","orgpu:PublicOrg_0"],"adrpost:streetAndNumber":"1 rue de la charité","org:sector":"Private","org:altName":[{"@language":"fr","@value":""}],"org:regNumber":"SCD11012015","adrpost:country":"http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR", "org:legalName":[{"@language":"fr","@value":"Société Coopérative Drômoise"}],"org:status":"Normal Activity","odisp:name":[{"@language":"fr","@value":"Société Coopérative Drômoise,Abbeville-Saint-Lucien (60480), France - France"}],"adrpost:postCode":"60480","adrpost:postName":"http://data.ozwillo.com/dc/type/geocifr:Commune_0/FR/FR-60/Abbeville-Saint-Lucien"},"causes":[],"location":"org.oasis.datacore.rest.server.resource.ResourceService.resourceToEntity(ResourceService.java:248)","message":"On resource http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/SCD11012015 in project org: Already exists at uri (forbidden in strict POST mode) :\nDCEntity[ http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/SCD11012015 (0) +types: [orgfr:Organisation_0, org:Organization_0, odisp:Displayable_0, relt:Target_0, adrpost:Postal_0, orgpu:PublicOrg_0] - {org:sector=Private, org:altName=[{v=, l=fr}], org:regNumber=SCD11012015, org:legalName=[{v=Société Coopérative Drômoise, l=fr}], org:status=Normal Activity, _ft=[{l=fr, v=[dromoise, societe, cooperative]}]} - 2016-01-11T09:12:49.277+01:00 ]","projectName":"org"}
| WARN | 09:16:50.047 | org.oasis_eu.spring.util.KernelLoggingInterceptor - POST request to https://data.ozwillo.com/dc/type/orgfr:Organisation_0 resulted in 400 Bad Request

On the creation request, the DC answers that it already exists. But I don't get why, when we search for it juste before asking for its creation, we have no results found.

When I try in the DC playground :

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 11, 2016

This last request returns no result because the organization has no country since it is indeed incomplete.
As said, try enabling full logging of requests to Datacore.

@bobeal
Copy link
Contributor Author

bobeal commented Jan 12, 2016

Finally, I was able to reproduce the problem in production with some (but not all) of my test accounts (and also with fresh new accounts). I was not able to reproduce it in pre-production, nor in development (even with fresh new accounts).

I attached a debugger to the portal and saw no double creation request.

Here is what the portal sent :

<DCResource{version=-1, baseUri='http://data.ozwillo.com/dc/type', type='orgfr:Organisation_0', iri='FR/TestOrgaAuDebug', uri='http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/TestOrgaAuDebug', encodedType='orgfr:Organisation_0', created=null, lastModified=null, createdBy='null', lastModifiedBy='null', values={adrpost:streetAndNumber=1 rue de la charité, org:type=null, org:phoneNumber=null, org:altName=[{@language=fr, @value=null}], org:regNumber=TestOrgaAuDebug, org:legalName=[{@language=fr, @value=TestOrgaAuDebug}], org:status=Normal Activity, org:email=null, odisp:name=[{@language=fr, @value=TestOrgaAuDebug,FrAin (88320), France - France}], adrpost:postCode=88320, adrpost:postName=http://data.ozwillo.com/dc/type/geocifr:Commune_0/FR/FR-88/FrAin, adrpost:cedex=null, adrpost:POBox=null, adrpost:supField=null, org:activity=null, org:sector=Private, org:webSite=null, adrpost:country=http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR, orgpu:officialId=null}},{Content-Type=[application/json], X-Datacore-Project=[org]}>

And the response received from the DC :

<201 Created,DCResource{version=0, baseUri='http://data.ozwillo.com/dc/type', type='orgfr:Organisation_0', iri='FR/TestOrgaAuDebug', uri='null', encodedType='null', created=2016-01-12T15:06:04Z, lastModified=2016-01-12T15:06:04.903Z, createdBy='606fc12f-cfe4-45d7-a49b-769c91b5ef57', lastModifiedBy='606fc12f-cfe4-45d7-a49b-769c91b5ef57', values={org:sector=Private, org:altName=[{@value=, @language=fr}], org:regNumber=TestOrgaAuDebug, @type=[orgfr:Organisation_0, org:Organization_0, odisp:Displayable_0, relt:Target_0, adrpost:Postal_0, orgpu:PublicOrg_0], org:legalName=[{@value=TestOrgaAuDebug, @language=fr}], org:status=Normal Activity}},{Date=[Tue, 12 Jan 2016 15:06:08 GMT], Content-Type=[application/json], Transfer-Encoding=[chunked], Server=[Ozwillo Datacore], Strict-Transport-Security=[max-age=31536000; includeSubDomains]}>

As it is not reproducible with all user accounts, maybe it has something to do with some user profile data (but which ?) ?

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 12, 2016

OK, it seems the code of the Portal itself is alright. So the problem is either in the portal REST layer (when the Portal DCResource is converted to JSON) or in the Datacore. As you guess, I incline to the former, especially if it has to do with profile data. But let's debug that together anyway.

@bobeal
Copy link
Contributor Author

bobeal commented Jan 15, 2016

I've just made an organization's creation test and I still have the problem with the geo-related fields not being persisted :

{
   "@id" : "http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/testAutreOrgaRemplie",
   "o:version" : 0,
   "@type" : [
      "orgfr:Organisation_0",
      "org:Organization_0",
      "odisp:Displayable_0",
      "relt:Target_0",
      "adrpost:Postal_0",
      "orgpu:PublicOrg_0"
   ],
   "dc:created" : 2016-01-15T16:25:24.000+01:00,
   "dc:modified" : 2016-01-15T16:25:24.180+01:00,
   "dc:creator" : c410d537-56fd-4db0-8cd1-fb730c68cf10,
   "dc:contributor" : c410d537-56fd-4db0-8cd1-fb730c68cf10,
   "org:activity" : "http://data.ozwillo.com/dc/type/orgactfr:Activit%C3%A9NAF2008_0/FR/01.45Z",
   "org:sector" : Public,
   "org:type" : SARL,
   "org:phoneNumber" : 0101010101,
   "org:webSite" : www.ozwillo.com,
   "org:altName" : [
      {
         "v" : TAOR,
         "l" : en
      }
   ],
   "orgpu:jurisdiction" : "http://data.ozwillo.com/dc/type/geocifr:Commune_0/FR/FR-93/Villepinte",
   "org:regNumber" : testAutreOrgaRemplie,
   "org:legalName" : [
      {
         "v" : Test Autre Orga Remplie,
         "l" : en
      }
   ],
   "org:status" : Normal Activity,
   "org:email" : [email protected],
   "orgpu:officialId" : 12345
}

@bobeal
Copy link
Contributor Author

bobeal commented Jan 15, 2016

PS : but it is still OK with my "old" account that did not had the problem

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 15, 2016

Could you try again now the "modify" case ? I've just found another problematic case, and tried something simpler that solves it (however, the "create" case should still be a problem with adrpostci fields).

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 15, 2016

I've been able to reproduce most of these problems.
I've just deployed in dev & preprod a simple, possibly comprehensive solution.

@mdutoo
Copy link
Collaborator

mdutoo commented Jan 18, 2016

Now that this new fix has been deployed in production, can you confirm it is OK ?

@bobeal
Copy link
Contributor Author

bobeal commented Jan 18, 2016

This use case is OK in prod (tested by @OzBruno and me).

But we have a problem when « acquiring » an existing organization (I gonna create a new issue for it).

@OzBruno
Copy link

OzBruno commented Jan 20, 2016

I've encountered again the problem with a different behaviour : I've created organisations on "My Network", coming back later I can modify the information like the address, but the org:country is still not stored in the DC whereas the adrpost:country is stored. And this prevents users to install some application such as Agrilocal.

@OzBruno OzBruno reopened this Jan 20, 2016
@mdutoo
Copy link
Collaborator

mdutoo commented Jan 20, 2016

It works alright for my test user in the Playground both for creation and modification:
/dc/type/orgfr:Organisation_0/FR/testImportMdu101
/dc/type/orgfr:Organisation_0/FR/testImportMdu100

So it is probably not the same problem as before but another one.

Indeed, looking again at what the portal sends (log provided by @bobeal ):

<DCResource{version=-1, baseUri='http://data.ozwillo.com/dc/type', type='orgfr:Organisation_0', iri='FR/TestOrgaAuDebug', uri='http://data.ozwillo.com/dc/type/orgfr:Organisation_0/FR/TestOrgaAuDebug', encodedType='orgfr:Organisation_0', created=null, lastModified=null, createdBy='null', lastModifiedBy='null', values={adrpost:streetAndNumber=1 rue de la charité, org:type=null, org:phoneNumber=null, org:altName=[{@language=fr, @value=null}], org:regNumber=TestOrgaAuDebug, org:legalName=[{@language=fr, @value=TestOrgaAuDebug}], org:status=Normal Activity, org:email=null, odisp:name=[{@language=fr, @value=TestOrgaAuDebug,FrAin (88320), France - France}], adrpost:postCode=88320, adrpost:postName=http://data.ozwillo.com/dc/type/geocifr:Commune_0/FR/FR-88/FrAin, adrpost:cedex=null, adrpost:POBox=null, adrpost:supField=null, org:activity=null, org:sector=Private, org:webSite=null, adrpost:country=http://data.ozwillo.com/dc/type/geocofr:Pays_0/FR, orgpu:officialId=null}},{Content-Type=[application/json], X-Datacore-Project=[org]}>

=> indeed we can see adrpost fields, that the Datacore unduly ate before. But we can't see the org:country field. Therefore it is a portal problem this time, see ozwillo/ozwillo-portal#363 .

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants