RealTime: "Errno::EPIPE: Broken pipe" "Async::Task client keep-alive failed" fails silently

[katsuya94]

We ran into an issue with slack-ruby-client (0.14.2) where after connecting and receiving messages for a short period of time. It stopped receiving messages entirely. Around the time it stopped receiving messages I was able to recover these logs.

                 |     /usr/local/bundle/gems/async-1.17.1/lib/async/task.rb:204 in `block in make_fiber'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/concurrency/async.rb:38 in `block (2 levels) in start_reactor'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/client.rb:128 in `run_ping!'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/client.rb:121 in `keep_alive?'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/api/ping.rb:11 in `ping'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/client.rb:187 in `send_json'"
                 |     /usr/local/bundle/gems/slack-ruby-client-0.14.2/lib/slack/real_time/socket.rb:22 in `send_data'"
                 |     /usr/local/bundle/gems/websocket-driver-0.7.0/lib/websocket/driver.rb:113 in `text'"
                 |     /usr/local/bundle/gems/websocket-driver-0.7.0/lib/websocket/driver/hybi.rb:191 in `frame'"
                 |     /usr/local/bundle/gems/websocket-driver-0.7.0/lib/websocket/driver/hybi.rb:228 in `send_frame'"
                 |     /usr/local/bundle/gems/async-websocket-0.8.0/lib/async/websocket/connection.rb:93 in `write'"
                 |     /usr/local/bundle/gems/async-io-1.23.0/lib/async/io/generic.rb:148 in `write'"
                 |     /usr/local/bundle/gems/async-io-1.23.0/lib/async/io/generic.rb:54 in `block in wrap_blocking_method'"
                 |     /usr/local/bundle/gems/async-io-1.23.0/lib/async/io/generic.rb:208 in `async_send'"
                 |     /usr/local/lib/ruby/2.5.0/openssl/buffering.rb:388 in `write_nonblock'"
                 |   → /usr/local/lib/ruby/2.5.0/openssl/buffering.rb:388 in `syswrite_nonblock'"
                 |   Errno::EPIPE: Broken pipe"
 5813m10s    error: <Async::Task:0x2ab09f0d86b0 client keep-alive failed> [pid=1]"
 {""type"":""ping"",""id"":5670}"
 {:type=>""ping"", :id=>5670}"

The client never attempted to reconnect, nor did it exit.

We are using also using Slack::RealTime::Client#started? as a health check, but the health check was NOT failing.

The concurrency library we're using is async-websocket.

Potentially relevant. We're using the websocket_proxy configuration option.

Potentially related: #257

Notably, 0.14.2 includes this commit 8e2df24

[dblock]

Definitely related to #257.

This looks like the keep alive worker itself failing and we're not handling that? I would appreciate any PRs that move this forward. In this case I think we're just missing a catch all for Errno::EPIPE that prevents the ping worker from dying.