WIT Deployments API fails with 401 to /api/user/services

[sbose78]

curl --header "Authorization: Bearer $token" https://prod-preview.openshift.io/api/deployments/spaces/f1432a96-179b-4d04-8abd-32e8972a950b

fails with

{"errors":[{"code":"unknown_error","detail":"failed to GET /api/user/services due to status code 401","status":"500","title":"Unknown error"}]}

https://github.com/fabric8-services/fabric8-wit/blob/master/controller/deployments_osioclient.go#L122

Note, call to wit/api/user/services

curl --header "Authorization: Bearer $token" https://api.prod-preview.openshift.io/api/user/services

works perfectly.

[joshuawilson]

related to #4621

[sbose78]

@tinakurian Could you please investigate and see if you can catch something ? As a starting point, setting up everything locally might be hard. However, verify if the token is being passed on to the /api/user/services call.

[joshuawilson]

I can confirm that it is not in the UI as I get no error when I point the UI at the prod server.

[jarifibrahim]

The failure is reproducible only on prod-preview. The deployments API works fine on prod.
@sbryzak could fabric8-services/fabric8-auth#689 have caused this?

[sbose78]

The request being passed on from the WIT Deployments API to WIT user/services API doesn't pass on the Authorization header.

{"err":"[Gwrp9CdH] 401 jwt_security_error: missing header \"Authorization\"","err_type":{},"error_message":"[Gwrp9CdH] 401 jwt_security_error: missing header \"Authorization\"","file":"/tmp/go/src/github.com/fabric8-services/fabric8-wit/jsonapi/jsonapi_utility.go","func":"github.com/fabric8-services/fabric8-wit/jsonapi.ErrorToJSONAPIError","level":"error","line":43,"msg":"an error occurred in our api","pid":1,"pkg":"jsonapi","req_headers":{"Accept-Encoding":["gzip"],"Forwarded":["for=174.129.114.152;host=prod-preview.openshift.io;proto=https;proto-version="],"Referer":["http://prod-preview.openshift.io/api/user/services"],"User-Agent":["Go-http-client/1.1"],"X-Forwarded-For":["174.129.114.152"],"X-Forwarded-Host":["prod-preview.openshift.io"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Request-Id":["Ojy7OwfpIg-47066"]},"req_id":"Ojy7OwfpIg-47066","time":"2018-12-11  11:04:47"}

[jarifibrahim]

I tried to reproduce the error locally but the header was correctly passed to the /user/service endpoint.

[jarifibrahim]

Interestingly I can no longer reproduce the issue.

➜  fabric8-wit git:(master) ✗ curl --header "Authorization: Bearer $token" https://prod-preview.openshift.io/api/deployments/spaces/f1432a96-179b-4d04-8abd-32e8972a950b | jq 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   127  100   127    0     0    127      0  0:00:01  0:00:01 --:--:--    95
{
  "data": {
    "attributes": {
      "applications": [],
      "name": "sbosedecspace1"
    },
    "id": "f1432a96-179b-4d04-8abd-32e8972a950b",
    "type": "space"
  }
}

UPDATE - I couldn't reproduce the issue because we moved back 2 versions of core on prod-preview. The latest core might still have this issue.

[joshuawilson]

I think this issue has to do with a difference in GoLang versions

[sbose78]

• The issue WIT Deployments API fails with 401 to /api/user/services #4631 started appearing from this commit fabric8-services/fabric8-wit@6f27f8b , so as a response @corinnekrych was considering reverting this remove flag to not run test in parallel fabric8-services/fabric8-wit#2375.
• However the code changes in that commit looks harmless.
• After some careful review, we found that prior to that commit, go1.9.4 was used https://ci.centos.org/view/Devtools/job/devtools-fabric8-wit-build-master/612/consoleFull . For all subsequent builds, go1.10 was being used https://ci.centos.org/view/Devtools/job/devtools-fabric8-wit-build-master/614/consoleFull This is the version golang used to run the go commands during build.
• There was no explicit specification of the version change by us anywhere that could have caused this change and hence we were considering making the golang download code flow simpler Remove USE_GO_VERSION_FROM_WEBSITE flag from dockerfile fabric8-services/fabric8-wit#2376

With zero code changes in the WIT deployments API , the above golang version changes is our prime suspect. The specific code here makes a call from WIT to another WIT API using the goa WIT client, and in the process doesn't pass headers properly causing a 401. This code needs to be refactored : WIT needs to call Tenant service instead of WIT --> WIT --> Tenant.

[sbose78]

Here's what we are gonna do next:

• The updated version Golang 1.10 got used instead of 1.9.4 because centos pulled out golang and the fallback in the Dockerfile.builder installed Golang 1.10 from the website. This will be fixed as part of of the work being done on Switch go builds from epel-testing to epel stable  #4618
• The UI components consuming this API is being deprecated.
• Till then, we are gonna stick to the older deployment of WIT ( which we have in prod and prod-preview today ). @michaelkleinhenz let's not deploy WIT till the above is done.
• On a sidenote, @jarifibrahim & @corinnekrych will work on moving WIT ( and other repos ) to 1.11 as being done for all other repos. (PR Use golang 1.11 from epel fabric8-services/fabric8-wit#2378)

[kwk]

• On a sidenote, @jarifibrahim & @corinnekrych will work on moving WIT ( and other repos ) to 1.11 as being done for all other repos.

Since when is Go 1.11 available in CentOS or RHEL?

[corinnekrych]

• On a sidenote, @jarifibrahim & @corinnekrych will work on moving WIT ( and other repos ) to 1.11 as being done for all other repos.

Since when is Go 1.11 available in CentOS or RHEL?

See Shoubhik link #4618

[jarifibrahim]

Here's what we'll be doing to investigate the failure.

What we know till now

• The /deployments/spaces/spaceID API on WIT calls the/api/user/services endpoint on WIT (wit to wit api call)[0]. The /deployments/spaces/spaceID API receives the Authorization header but the /user/services endpoint doesn't receive the Authorization header and the JWT middleware fails with missing header error[1] even before the request reaches the user service controller. The JWT middleware returns the 401 error.

• The issue is reproducible only on prod preview. We have tried locally with multiple versions of golang and the docker image that runs on prod preview but we couldn't reproduce the error locally.

What we are going to do

1. We will be using go1.9.4 in the builder container and ensure the /deployment API works fine. This build will be our fall back incase things go south. The go version is our prime suspect right now. We think the version change has caused this failure. See also WIT Deployments API fails with 401 to /api/user/services #4631 (comment).
   PR Use golang 1.9.4 from website fabric8-services/fabric8-wit#2383
2. Update prod-preview to use go1.11. The /deployments/spaces API should start failing. We will be pushing more log statements so that we can figure out why the header was removed. The e2e tests would fail while we are investigating the failure.

cc @corinnekrych @sbose78
[0] - https://github.com/fabric8-services/fabric8-wit/blob/master/controller/deployments_osioclient.go#L106
[1] - https://github.com/fabric8-services/fabric8-wit/blob/master/goamiddleware/jwt_middleware.go#L92

[sbose78]

👍 CC @ppitonak