Debugging application startup problems

Sometimes an application refuses to startup. When this happens, Phusion Passenger reports one of these messages:

An error occurred while starting up the preloader. It exited before signalling successful startup back to Phusion Passenger.
An error occurred while starting up the preloader: it did not write a startup response in time.
An error occurred while starting the web application: it did not write a handshake message in time.
An error occurred while starting the web application. It exited before signalling successful startup back to Phusion Passenger.

Phusion Passenger reports this error if the application did not finish initializing within a time limit, or if it exited without sending Phusion Passenger a message that says "I've initialized successfully!".

The spawning process explained

To understand what's going on, consider what Phusion Passenger does under the hood. Phusion Passenger spawns an application by running one of the following command:

• If your shell is bash: bash -l -c '/path-to/SpawnPreparer /path-to-loader-or-preloader'
• If your shell is not bash: /path-to/SpawnPreparer /path-to-loader-or-preloader

So when launching your application, there are at most 4 components at work:

1. If your shell is bash, then everything else is executed through bash, by passing the -l (login shell) flag to bash. This causes bash to load its startup files, e.g. bashrc, profile, etc. The reason why we do this is because a lot of users try to set environment variables in their bashrc, and they expect these environment variables to be picked up by applications spawned by Phusion Passenger. Unfortunately environment variables don't work that way, but we support it anyway because it is good for usability.
2. Bash immediately invokes the Phusion Passenger SpawnPreparer executable. Or, if the user's shell is not bash, SpawnPreparer is invoked directly. This executable is responsible for setting certain environment variables, current working directory, and other process environmental conditions. When SpawnPreparer is done, it executes the loader or the preloader.
3. If passenger_spawn_method is set to smart (the default), and there is a preloader available for the application's programming language, then this step executes the language-specific preloader. If either of the previous conditions are not met (and thus the passenger_spawn_method is automatically forced to direct), then this step executes the language-specific loader.
   • A preloader loads the application code in memory, and forks off worker processes every time Phusion Passenger needs another application process, which is how the "smart" spawning method is implemented.
   • A loader also loads the application code. But unlike the preloader it does not fork off worker processes. Every time Phusion Passenger needs a process, it starts another loader that fully loads the application code.

This is documented in detail in Spawning methods explained. 4. The loader or preloader loads your application by evaluating its startup file (Ruby: config.ru; Python: passenger_wsgi.py; Node.js: app.js).

In any of the above 4 steps, something can go wrong. In versions prior to Phusion Passenger 4, and in other application servers such as WEBrick or (G)Unicorn, steps 1 and 2 don't exist, which is why your application may startup correctly under those servers.

Possible causes of problems

Stdout redirection

Phusion Passenger uses the application's stdout for communication with the application. This means that if, during any of those steps, stdout is closed, overwritten or redirected to a file, then Phusion Passenger loses its means to communicate with the application. After a while, Phusion Passenger concludes that the application fails to start up, and reports an error.

It is however safe to write data to stdout as long as the data does not begin with !> . Phusion Passenger uses !> to mark internal communication messages.

Solution for this problem: ensure that your application and your bash startup files never redirect stdout to a file, or otherwise modify that file descriptor.

To test whether something along the spawning chain messed with stdout, add some simple code to the beginning of your startup file which performs two things:

• It prints a message to stdout.
• It creates a file.

Then try to start the application through Phusion Passenger, and verify:

• whether the file exists.
• whether the message appears in your web server error log (Phusion Passenger redirects all application output to that log).

If the file exists, then apparently your application did start up, but something messed with stdout.

Early termination in bash

Some bash startup files contain code that cause bash to terminate if there is no TTY. Phusion Passenger always spawns applications without a TTY. A typical example of such a code is:

if ! tty -s; then exit; fi

Solution 1 for this problem: remove such code from your bash startup files.

To test whether your bash startup files are the culprit, add a line to the end of your bashrc that creates a file, then try to start your application through Phusion Passenger. If the file does not exist, then apparently some earlier code in the bash startup files caused bash to exit.

Solution 2 for this problem: disable the loading of bashrc.

Since version 4.0.20, you can set PassengerLoadShellEnvvars off (Apache) or passenger_load_shell_envvars off (Nginx) to disable the loading of bashrc.

Application startup freeze

The application itself may freeze during startup for any reason. It might be loading a huge file that takes forever, it might be frozen while trying to access an NFS share that is down, it might be trying to download a file from S3 during network problems, or it might have an infinitely looping algorithm somewhere.

Solution for this problem: isolate the offending code in your application. The code may be in a library. The easiest strategy is to put print statements in multiple places in the code, and check which print messages appear in the web server error log (where Phusion Passenger redirects all application output to). If you do not see certain print output, then you know something has gone wrong between the last print statement and the current one. Keep adding print statements until you've isolated the offending part.

Server too busy

It could also be that your server is so busy doing something (either CPU-wise or disk-wise) that it fails to start an application process within a reasonable amount of time. The default startup limit is 90 seconds.

Solution for this problem: Change the timeout with one of the following configuration options:

• Apache: PassengerStartTimeout <SECONDS>
• Nginx: passenger_start_timeout <SECONDS>

These configuration options have been introduced in version 4.0.15.