[Solved] Reverb not dispatching events in production

[WesWeCan]

Reverb Version

@beta (out of the box after install)

Laravel Version

11.0

PHP Version

8.3.4

Description

In a local environment (non herd) with reverb:start when a event is dispatched I get the message in my console log. But in production on a Plesk server it does not. No error messages or anything in the logs that seem to point at the solution.

Steps To Reproduce

Create a new laravel installation, install broadcast.

local .env

REVERB_APP_ID=271094
REVERB_APP_KEY=esa4d8rnyeyadiexuram
REVERB_APP_SECRET=pyubfujwpiqrdwp60zhu
REVERB_HOST="localhost"
REVERB_PORT=8080
REVERB_SCHEME=http

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

production .env

REVERB_APP_ID=271094
REVERB_APP_KEY=esa4d8rnyeyadiexuram
REVERB_APP_SECRET=pyubfujwpiqrdwp60zhu
REVERB_HOST="sub.domain.com"
REVERB_PORT=443
REVERB_SCHEME=https

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

In a controller:

class SampleController extends Controller
{
    public function store(Request $request){
        $message = $request->input('message');
        SampleEvent::dispatch($message);
        return response()->json(['success' => true, 'message' => 'Message sent successfully']);
    }
}

The Event

class SampleEvent implements ShouldBroadcastNow
{
    use Dispatchable, InteractsWithSockets, SerializesModels;

    public $message;

    /**
     * Create a new event instance.
     */
    public function __construct($message)
    {
        $this->message = $message;
    }

    /**
     * Get the channels the event should broadcast on.
     *
     * @return array<int, \Illuminate\Broadcasting\Channel>
     */
    public function broadcastOn(): Channel
    {
        return new Channel('sample');
    }

    public function broadcastAs(): string
    {
        return 'message.sent';
    }
}

On the front-end (works local but not in production)

  window.Echo.channel('sample')
        .listen('.message.sent', (e) => {
            console.log(e);
        })

On the server I have added 'additional nginx directives'

location /app {
	proxy_http_version 1.1;
	proxy_set_header Host $http_host;
	proxy_set_header Scheme $scheme;
	proxy_set_header SERVER_PORT $server_port;
	proxy_set_header REMOTE_ADDR $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "Upgrade";

	proxy_read_timeout 300s;
	proxy_connect_timeout 75s;

	proxy_pass http://0.0.0.0:8080;
}

A connection on the front-end is made over wss://sub.domain.com/app/~~~~
In the debug of reverb I see a subscribe message to the channel:

1▕ {
   2▕     "event": "pusher:subscribe",
   3▕     "data": {
   4▕         "auth": "",
   5▕         "channel": "sample"
   6▕     }
   7▕ }

But the event doesn't trigger the messages being send in production, but they do in local development.

It feels like that the Laravel Framework is not able to send the message to the Reverb server for unknown reasons.

[WesWeCan]

I have tested this issue with the response #107 (comment) here, but this was not the solution.

[dysTOS]

I have the same problem https://github.com/laravel/reverb/issues/112 here, seems its also pretty much the same conditions. Laravels Broadcasting system cannot connect to the reverb-server, still found no solution.

[WesWeCan]

I have the same problem https://github.com/laravel/reverb/issues/112 here, seems its also pretty much the same conditions. Laravels Broadcasting system cannot connect to the reverb-server, still found no solution.

I looked into your issue #112 but in this case no exception is thrown. I checked the logs and checked this with Telescope.

[WesWeCan]

I have tried to implement the solution of #107 (comment) with no success. Echo connects to Reverb and a subscribed message is logged in --debug but when a event is dispatched no socket messages are send in production, I have tried with queue:listen as well.

These two issues seem related but are different.

To me it seems that the issue is somewhere between Laravel Broadcast communicating with Reverb. But no exceptions or logs are made.

What I noticed is that in Telescope there are 0 listeners. It feels unrelated but worth mentioning. This is the same for local as well as production.

Is there still an experiment that I can conduct?

[joedixon]

Hi @WesWeCan, the backend routes for Reverb are prefixed with /apps and not /app. Your location rule in your Nginx configuration, is prefixed /app only, so that route will never be hit.

[WesWeCan]

Hi @joedixon Thanks for your reply. But adding a /apps to NGINX doesn't do the trick.

location /app {
	proxy_http_version 1.1;
	proxy_set_header Host $http_host;
	proxy_set_header Scheme $scheme;
	proxy_set_header SERVER_PORT $server_port;
	proxy_set_header REMOTE_ADDR $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "Upgrade";

	proxy_pass http://0.0.0.0:8080;
}

location /apps {
	proxy_http_version 1.1;
	proxy_set_header Host $http_host;
	proxy_set_header Scheme $scheme;
	proxy_set_header SERVER_PORT $server_port;
	proxy_set_header REMOTE_ADDR $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "Upgrade";

	proxy_pass http://0.0.0.0:8080;
}

Also leaving this here for people that stumble on this same problem in the future.

[amypo-tech]

I have the same type of issue. nothing was helpful.

implementing web-sockets in production is like finding water in mars.

All the possible ways i have tried.

1. all the possible .env configurations
2. nginx reverse proxy configuration
3. config/reverb.php config changes with and without certs and different ports

if some one gives a proper setup or guide from start to end would help for production.

in local it took only few minutes to implement. this is the same thing happened before launch of reverb. waiting nearly 6 months to implement in production

[WesWeCan]

I have the same type of issue. nothing was helpful.

implementing web-sockets in production is like finding water in mars.

All the possible ways i have tried.

1. all the possible .env configurations
2. nginx reverse proxy configuration
3. config/reverb.php config changes with and without certs and different ports

if some one gives a proper setup or guide from start to end would help for production.

in local it took only few minutes to implement. this is the same thing happened before launch of reverb. waiting nearly 6 months to implement in production

What kind of environment are you using?
Also Plesk with Apache & NGINX?

[dominik-eller]

Having the same error using Laravel Forge.
I am getting "invalid message format" when subscribing to a channel

and when I notify a user via broadcast I am getting a internal server error:

production.ERROR: Pusher error: Internal server error.. {"exception":"[object] (Illuminate\\Broadcasting\\BroadcastException(code: 0): Pusher error: Internal server error.. at /home/***/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:164)

EDIT:
I deactivated and activated Laravel Reverb in die Laravel Forge Application Panel and restarted nginx and it is working now. Maybe the nginx config for the reverse proxy was not set correctly or nginx was not reloaded automatically.

The working configuration in my .env. file:

REVERB_APP_ID=***
REVERB_APP_KEY=***
REVERB_APP_SECRET=***
REVERB_HOST=127.0.0.1
REVERB_PORT=8080
REVERB_SCHEME=http

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST=ws.domain.tld
VITE_REVERB_PORT=443
VITE_REVERB_SCHEME=https

[WesWeCan]

@joedixon Okay! I figured it out. It is a really stupid thing that I missed since I was too much focussed on the Reverb side of this. Maybe you have the same oversight @amypo-tech?

In short the issue was just a wrong .env config.

In my production .env I had

BROADCAST_CONNECTION=log
FILESYSTEM_DISK=local
QUEUE_CONNECTION=database

but it should be

BROADCAST_CONNECTION=reverb
FILESYSTEM_DISK=local
QUEUE_CONNECTION=database

I am not sure if this is set by the install of reverb or that I did it locally.

So for people that stumble upon this same issue (according to google, I'm not the only one). I always find it frustrating if there is no complete solution given when I google myself.

in .env

...
BROADCAST_CONNECTION=reverb
FILESYSTEM_DISK=local
QUEUE_CONNECTION=database
...
REVERB_SERVER_HOST=127.0.0.1
REVERB_SERVER_PORT=8080

REVERB_APP_ID=271094
REVERB_APP_KEY=esa4d8rnyeyadiexuram
REVERB_APP_SECRET=pyubfujwpiqrdwp60zhu
REVERB_HOST="sub.domain.com"
REVERB_PORT=443
REVERB_SCHEME=https

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

For me in PLESK (apache + NGINX)
Proxy mode = ON

Additional NGINX Directives:

location /app {
	proxy_http_version 1.1;
	proxy_set_header Host $http_host;
	proxy_set_header Scheme $scheme;
	proxy_set_header SERVER_PORT $server_port;
	proxy_set_header REMOTE_ADDR $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "Upgrade";

	proxy_pass http://0.0.0.0:8080;
}

location /apps {
	proxy_http_version 1.1;
	proxy_set_header Host $http_host;
	proxy_set_header Scheme $scheme;
	proxy_set_header SERVER_PORT $server_port;
	proxy_set_header REMOTE_ADDR $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "Upgrade";

	proxy_pass http://0.0.0.0:8080;
}

---

Running the Reverb server automatically

This may differ in your environment but this is now tested in Plesk AlmaLinux 8, PHP 8.3. All on your own risk form here!
In my case I have the reverb server as a sub-domain.

yum install supervisor

nano /etc/supervisord.conf

[program:reverb-worker]
process_name=%(program_name)s_%(process_num)02d
command=/opt/plesk/php/**8.3**/bin/php /var/www/vhosts/DOMAIN.COM/SUB.DOMAIN.COM/artisan reverb:start
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/vhosts/DOMAIN.COM/logs/reverb-worker.log

When updating the .env you should do
systemctl restart supervisord

[Hiiippo]

location /app {
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Scheme $scheme;
proxy_set_header SERVER_PORT $server_port;
proxy_set_header REMOTE_ADDR $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";

proxy_pass http://0.0.0.0:8080;
}

location /apps {
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Scheme $scheme;
proxy_set_header SERVER_PORT $server_port;
proxy_set_header REMOTE_ADDR $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";

proxy_pass http://0.0.0.0:8080;
}

@WesWeCan This fixed it for me. Thank you, you're a godsend!

[ritwikBiswas007]

I am adding this to cloud admin but not working

`server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
{{ssl_certificate_key}}
{{ssl_certificate}}
server_name neo.zepay.money;
{{root}}

{{nginx_access_log}}
{{nginx_error_log}}

if ($scheme != "https") {
rewrite ^ https://$host$uri permanent;
}

location ~ /.well-known {
auth_basic off;
allow all;
}

{{settings}}

location / {
{{varnish_proxy_pass}}
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_hide_header X-Varnish;
proxy_redirect off;
proxy_max_temp_file_size 0;
proxy_connect_timeout 720;
proxy_send_timeout 720;
proxy_read_timeout 720;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
proxy_temp_file_write_size 256k;
}

location /app {
    proxy_pass http://127.0.0.1:8081;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location ~* ^.+.(css|js|jpg|jpeg|gif|png|ico|gz|svg|svgz|ttf|otf|woff|woff2|eot|mp4|ogg|ogv|webm|webp|zip|swf|map)$ {
add_header Access-Control-Allow-Origin "*";
expires max;
access_log off;
}

if (-f $request_filename) {
break;
}
}

server {
listen 8080;
listen [::]:8080;
server_name neo.zepay.money;
{{root}}

try_files $uri $uri/ /index.php?$args;
index index.php index.html;

location ~ .php$ {
include fastcgi_params;
fastcgi_intercept_errors on;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
try_files $uri =404;
fastcgi_read_timeout 3600;
fastcgi_send_timeout 3600;
fastcgi_param HTTPS "on";
fastcgi_param SERVER_PORT 443;
fastcgi_pass 127.0.0.1:{{php_fpm_port}};
fastcgi_param PHP_VALUE "{{php_settings}}";
}

location ~* ^.+.(css|js|jpg|jpeg|gif|png|ico|gz|svg|svgz|ttf|otf|woff|woff2|eot|mp4|ogg|ogv|webm|webp|zip|swf|map)$ {
add_header Access-Control-Allow-Origin "*";
expires max;
access_log off;
}

if (-f $request_filename) {
break;
}
}`

[vw5921321]

for some reason, after switching from http to https, my pusher server does not broadcast the event, i've applied the configs as mentioned by WesWeCan, but its still the same.
ive found out that for some reason vendor/laravel/reverb/src/Protocols/Pusher/EventDispatcher.php line 45 :
if (! $channel = app(ChannelManager::class)->for($app)->find($channel)) {
the channel manager all() here was returning empty array.

i've changed it to
if (! $channel = app(ChannelManager::class)->for($app)->findOrCreate($channel)) {

which then worked, the events are broadcasted and shown on client and in reverb console and with --debug (previously the clients did subscribe to the channel before the event was broadcasted, just to clarify)

after doing some more checks, i couldn't figure out why the channel manager was returning empty channels array in the first place, and after reverting back to find() and restarting the reverb server, the events were still being broadcasted successfully.
no idea what's the issue.

update : woops! i just realized that i wasn't subscribing to the same channel as the event previously! you need to make sure the channel the event broadcasts to is subscribed to by anything, before broadcasting the event! otherwise the event wouldn't even be broadcasted from EventDispatcher, and you wouldn't see anything on client side and on reverb console.