Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

logstash input plugin TLS port hangs causes denial of service (DOS) #77

Open
jmschmaus opened this issue Jul 12, 2017 · 9 comments
Open

logstash input plugin TLS port hangs causes denial of service (DOS) #77

jmschmaus opened this issue Jul 12, 2017 · 9 comments
Labels
bug

Comments

@jmschmaus
Copy link

We want to move elastic/logstash#7650 to logstash-input-tcp, which I didn't know about when I opened the ticket.

Here is text from #7650:

Logstash 5.3 uses jruby-openssl 0.9.16 for SSL/TLS. jruby-openssl SSLSocket.java does SSL handshake. SSLSocket will not return until the SSL handshake is complete. Logstash will not do another accept on the TLS socket until SSL handshake is complete. If a non-TLS client connects to logstash's TLS port, the port is hung and unusable for any other clients.

For all general issues, please provide the following details for fast resolution:

Version: 5.3.0
Operating System: CentOS 6
Config File (if you have sensitive info, please remove it): Please see below for TLS configuration
Sample Data:
Steps to Reproduce:

  1. Configure ssl for logshipper tcp input:
input {
...
tcp {
host => "0.0.0.0"
type => "tcp_json_event"
port => "10059"
ssl_enable => true # This needs to be true for the other ssl parameters to be considered
ssl_verify => false # Don't validate the cert against the CA. Useful for self signed certs
ssl_cert => "ssl cert" # SSL Cert
ssl_key => "ssl key"
ssl_extra_chain_certs => "trusted cert" # CA certs
codec => json_lines{
charset => "ISO-8859-1"
}
...

  1. Verify TLS operation via openssl s_client -connect localhost:10059

  2. Open non-TLS connection to port 10059, configured for TLS:
    nc localhost 10059

  3. Verify SSL handshake no longer operational (step (2)).
@jmschmaus jmschmaus mentioned this issue Jul 12, 2017
Closed
@jmschmaus jmschmaus changed the title logstash input plugin TLS port hangs logstash input plugin TLS port hangs causes denial of service (DOS) Aug 7, 2017
@original-brownbear original-brownbear self-assigned this Sep 1, 2017
@jdlourenco
Copy link

jdlourenco commented Nov 15, 2017 •
edited
Loading

I am experiencing a similar issue, tough I am not sure it is exactly the same.

I have a tcp input with tls enabled which seems to work fine for some time.
When I try to connect using openss I usually get this:

openssl s_client -connect machine:9400 -CAfile config/certs/ca.hidra-prd-logstash.pem -cert config/certs/las.hidra-prd-logstash.crt -key config/certs/las.hidra-prd-logstash.key
CONNECTED(00000003)
depth=1 C = PT, ST = Lisboa, L = Default City, O = XXX, OU = DCY-CSD, emailAddress = [email protected]
verify return:1
depth=0 C = PT, ST = Lisboa, L = Default City, O = XXX, OU = DCY-CSD, CN = logstash, emailAddress = [email protected]
verify return:1
---
Certificate chain
 0 s:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/CN=logstash/[email protected]
   i:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/[email protected]
 1 s:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/[email protected]
   i:/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDgjCCAmoCCQDnQz+vbbVU6TANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJQ
VDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxDzANBgNV
BAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDEfMB0GCSqGSIb3DQEJARYQcHVs
c29AdGVsZWNvbS5wdDAeFw0xNzA5MjUxNDU5NTdaFw0yMDA3MTUxNDU5NTdaMIGM
MQswCQYDVQQGEwJQVDEPMA0GA1UECAwGTGlzYm9hMRUwEwYDVQQHDAxEZWZhdWx0
IENpdHkxDzANBgNVBAoMBkFsdGljZTEQMA4GA1UECwwHRENZLUNTRDERMA8GA1UE
AwwIbG9nc3Rhc2gxHzAdBgkqhkiG9w0BCQEWEHB1bHNvQHRlbGVjb20ucHQwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDidCg1cp9Jp4dVbMlWd7f1XQ4b
B8IjNP+FbfKyakccopvAaA+PkRG9iZ+VQjs+RNwjazleDDXIF2BPiWS7tMT0s6hw
XEeWVZ02E8X3oKA3NXz5Nv0j8p7Pzc95kkTR1o3fCVSIc+z9+5WCAn0k4jyKkj82
FMIkGPbcMlYBFBbYUsiH3YUwhrj6Zsdvm3qrR8al7rscrkKYLawn0v3reXNiTIwY
Q0Hpj821SQk+OqGJB2OZbApFlb0jdycdkggoWtQjVzRPT+bYaqZYztprBVIatt5J
yjRNZ3WPsDKMtem+df26BHr2+WyI3fMoWYIBQm4WOsAcmDaGwmKF7XpPqK9XAgMB
AAEwDQYJKoZIhvcNAQELBQADggEBAFjKLjgQovkZWsW61IMdxQOy1VHIJS79X2Iz
FteAXf0s1kdkFY0YV+5TDisEn0P2mOIrjQ2tWfJbJEiKuiJ4rdRUNGwpbV3Il8gL
YRxbxfXOSnBMQ7MYAxqv6ALsVAcG3aqKcq32cwSjXSWXvKERNlvDYFm5MgYrbibj
IynQ7KYWDJqHDWC4WsMFDFQHR2gaYvVZ6YfnVAaOlzpzAQ1ZM48Spl8S3kYSxnRi
W1i5HcU0IuPOTqJ7lCHDMTmJNSaCS3amwWJU0UCADfxkzNRKe+Eq70CQtzS8ZdvV
pVQF7sdtMJFUfyzpw0B5agoKuREt0I70Vm7EhiTgLNVdQv5f8Ao=
-----END CERTIFICATE-----
subject=/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/CN=logstash/[email protected]
issuer=/C=PT/ST=Lisboa/L=Default City/O=XXX/OU=DCY-CSD/[email protected]
---
No client certificate CA names sent
Server Temp Key: ECDH, secp521r1, 521 bits
---
SSL handshake has read 2500 bytes and written 2634 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA256
    Session-ID: 5A0C3B2DB5E145C12D67E8494675F1449F77073B1BB6DB7EBDCAC44322E21E8F
    Session-ID-ctx:
    Master-Key: 3C1DD3AABC322E3344B0D279B79B0A6579E83B9512005951303952A946F926BD7F1F25AAF13A3DCDB20DBD9D052512AE
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1510751021
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

However, after running for some time I am no longer able to establish a new connection as it hangs:

openssl  s_client -connect localhost:9400 -CAfile ca.pem -cert cert.crt -key cert.key
CONNECTED(00000003)

Listing all connections to my logstash instance I didn't find any non-TLS, and they all seemed to be working and sending data, so I'm not sure the issue is the same.

@jmschmaus were you able to connect again after closing the non-TLS connection or was it no longer able to establish any other new TLS connection after that?

@jmschmaus
Copy link
Author

jmschmaus commented Nov 15, 2017 via email

@jdlourenco
Copy link

Hi John,

That's exactly it! Thank you for the exhaustive description.
It's really useful for finding the culprit connection

This has a huge impact in the reliability of our infrastructure since we are relying on a TCP input for ingesting all of our data.

Any prediction on when this will be fixed?

Best regards,
José

@jmschmaus
Copy link
Author

jmschmaus commented Nov 15, 2017 via email

@original-brownbear
Copy link
Contributor

Looked over this a little today. There is no quick fix here it seems. This will have to wait until (and will be automatically fixed by moving the encrypted path to Netty like we did for the unencrypted path).

@original-brownbear original-brownbear removed their assignment Nov 16, 2017
@jdlourenco
Copy link

@original-brownbear any prediction on when this will be fixed?

@original-brownbear
Copy link
Contributor

@jdlourenco not for now no.

@ghost
Copy link

ghost commented Aug 30, 2018

@original-brownbear any prediction on when this will be fixed?

@sunilchadha1973
Copy link

Hi @original-brownbear,

Is this issue resolved in Logstash 7.x version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jdlourenco @original-brownbear @jmschmaus @sunilchadha1973