server.rewriteBasePath expects a trailing slash

[gfarrell]

Kibana version:
7.6.2 (docker)

Elasticsearch version:
N/A

Server OS version:

• Docker version 19.03.8, build afacb8b7f0
• Linux 4.15.0-91-generic

Browser version:
N/A

Browser OS version:
N/A

Original install method (e.g. download page, yum, from source, etc.):
docker-compose

Describe the bug:
As per SO question: nginx reverse-proxy with a base-path to kibana (/kibana), configuration below. Kibana has server.basePath and server.rewriteBasePath are set to kibana and true (see config below). When accessing my.host.com/kibana I get a JSON-encoded 404 error.

nginx.conf:

  location /kibana {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_set_header        Connection "Keep-Alive";
    proxy_set_header        Proxy-Connection "Keep-Alive";

    proxy_pass              http://kibana:5601/;
    proxy_read_timeout      90;
    proxy_redirect          http://kibana:5106 https://my.host.com/kibana;
    proxy_buffering         off;
  }

kibana.yml

elasticsearch.hosts: ["http://elasticsearch:9200"]
logging.dest: stdout
server.host: kibana
server.basePath: /kibana
server.rewriteBasePath: true

This was fixed by adding a rewrite rule for nginx and setting server.rewriteBasePath to false:

updated nginx.conf rule:

    rewrite                 /kibana/(.*)$ /$1 break;

Steps to reproduce:

1. Configure nginx and kibana as above
2. Try to access my.host.com/kibana
3. Get 404 error

Expected behavior:
Kibana interface should load normally

Screenshots (if relevant):
N/A

Errors in browser console (if relevant):
N/A

Provide logs and/or server output (if relevant):

kibana logs:

{"type":"response","@timestamp":"2020-03-29T22:36:12Z","tags":[],"pid":6,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"my.host.com","x-real-ip":"185.200.118.103","x-forwarded-for":"185.200.118.103","x-forwarded-proto":"https","connection":"Keep-Alive","proxy-connection":"Keep-Alive","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:74.0) Gecko/20100101 Firefox/74.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"172.28.0.10","userAgent":"172.28.0.10"},"res":{"statusCode":404,"responseTime":188,"contentLength":9},"message":"GET / 404 188ms - 9.0B"}
{"type":"error","@timestamp":"2020-03-29T22:36:12Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n    at emitWarning (internal/process/promises.js:97:15)\n    at emitPromiseRejectionWarnings (internal/process/promises.js:143:7)\n    at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"}
{"type":"error","@timestamp":"2020-03-29T22:36:12Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 7)","name":"UnhandledPromiseRejectionWarning","stack":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 7)"}
{"type":"response","@timestamp":"2020-03-29T22:36:13Z","tags":[],"pid":6,"method":"get","statusCode":404,"req":{"url":"/","method":"get","headers":{"host":"my.host.com","x-real-ip":"185.200.118.103","x-forwarded-for":"185.200.118.103","x-forwarded-proto":"https","connection":"Keep-Alive","proxy-connection":"Keep-Alive","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:74.0) Gecko/20100101 Firefox/74.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"172.28.0.10","userAgent":"172.28.0.10"},"res":{"statusCode":404,"responseTime":160,"contentLength":9},"message":"GET / 404 160ms - 9.0B"}
{"type":"error","@timestamp":"2020-03-29T22:36:13Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)\n    at emitWarning (internal/process/promises.js:97:15)\n    at emitPromiseRejectionWarnings (internal/process/promises.js:143:7)\n    at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"}
{"type":"error","@timestamp":"2020-03-29T22:36:13Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 12)","name":"UnhandledPromiseRejectionWarning","stack":"TypeError: Cannot read property 'statusCodes' of undefined\n    at Network._server.events.on (/usr/share/kibana/node_modules/oppsy/lib/network.js:56:38)\n    at module.exports.internals.Podium.internals.Podium.emit (/usr/share/kibana/node_modules/podium/lib/index.js:220:49)\n    at Request._finalize (/usr/share/kibana/node_modules/hapi/lib/request.js:406:27)\n    at Request._reply (/usr/share/kibana/node_modules/hapi/lib/request.js:345:14)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 12)"}

Any additional context:
N/A

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[mistic]

@jbudz do you have any idea here?

[mistic]

@gfarrell I got it to working locally. I haven't used a docker setup but instead a local kibana installation + local nginx.

I've used the following configurations:

nginx.conf

events {
    worker_connections  1024;
}

http {
    server {
        listen 8080;
        
        location /kibana {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_pass http://localhost:5601/kibana;
        }
    }
}

kibana.yml

elasticsearch.username: "kibana"
elasticsearch.password: "..........."
server.basePath: /kibana
server.rewriteBasePath: true

Could you try with that setup?

[gfarrell]

I think doing this outside of docker probably defeats the point as the bug might be specific to the way the kibana docker images work, or the way it behaves there. Would suggest doing it with a simple docker-compose setup (also because my docker setup is linux, but my main machine is a mac). I can write a basic docker-compose file if that helps you find the bug.

[gfarrell]

I have an idea of what might be wrong actually, although I think the documentation is extremely unclear for this. Will investigate.

[mistic]

@gfarrell let me know about your findings so we can improve the documentation if that is the case 😃

[jbudz]

This is usually caused by a missing trailing slash, but it's very strange. My understanding is probably limited but what I've seen:

Kibana expects a trailing slash on our routes. Browsers will automatically inject this, so it's not a intuitive configuration because nginx won't. With rewriteBasePath localhost:5601/ should be a sufficient target. Without, localhost:5601/(.*)$/

[Stevenpc3]

Did you intend to have this port mistake or was it intentional?

[gfarrell]

The issue was a good 3.5 years ago so I can no longer remember, and I imagine it has been subsequently fixed (I'm no longer using kibana for this). I'll close the issue.