All plugins return 0 for stats

[dogenpunk]

I'm current running a Rails 3.0.4 application with Passenger 3.0.7 on FreeBSD 8.2 with Munin 1.4.4. After installing this gem and moving the symlinks to /usr/local/etc I get nothing but 0 return values for every plugin except munin_passenger_status. I have installed the application specific plugins, which also return 0 for all statistics. When I run request-log-analyzer itself I get statistics as expected.

[dogenpunk]

Also, when running the app specific plugins directly I get 'yes' when run with autoconf and an error when run with debug:

Filepath unspecified. Exiting

I believe this is may be due to /proc/meminfo being specific to Lunixes.

[wakemaster39]

For the rails plugins at least the issue has to do with the logger format. The request-log-analyzer gem does its best to try and figure out which log format it is reading, but has issues determining between rails 2 and 3. It defaults to rails 2 I believe, which makes it run into errors in reading the log and you get out zero's. To fix this, be sure to set "env.log_format rails3" in the plugin-conf.d folder for EACH of the files that use this gem.

Also note that by default the generated files specify the binary location for ruby, and passeneger-memory stats, etc to be in the "/usr/local/bin/" folder, while if you install these gems as root, they will default to the "/usr/bin" folder. The defaults are the safer and better choice if you can make them install there, but you can edit all the config files to remove the local part and this should get your plugins running.

[dogenpunk]

Ok, thanks. Setting the log format worked for the request-log-analyzer plugins. The only one I seem to have an issue with now is the munin_passenger_memory_stats plugin. It's still returning 0 for memory.value. Is this to be expected? When I run

ruby munin_passenger_memory_stats debug

I do get the expected output from passenger-memory-stats.

[wakemaster39]

Can you paste the conf file for this? It is likely an issue there since running it from ruby is working.

[dogenpunk]

Here it is:

[munin_passenger_memory_stats]
user root
command /usr/local/bin/ruby %c
env.passenger_status /usr/local/bin/passenger-status
env.apache_pid_file /var/run/httpd.pid
env.passenger_memory_stats /usr/local/bin/passenger-memory-stats
env.graph_category Passenger

I've confirmed each path. Ruby is located in /usr/local/bin as I had to manually upgrade from 1.8 to 1.9.

[wakemaster39]

I believe the issue is that you don't have quotes on your variables.
env.passenger_status /usr/local/bin/passenger-status env.apache_pid_file /var/run/httpd.pid env.passenger_memory_stats /usr/local/bin/passenger-memory-stat

These should be formatted like
env.passenger_status '/usr/local/bin/passenger-status' env.apache_pid_file '/var/run/httpd.pid' env.passenger_memory_stats '/usr/local/bin/passenger-memory-stat'

Its the only issue that I can see that differ from my commands.

[dogenpunk]

I gave this a try and I still get the same results. Also, none of the other plugins have the values quoted in this manner. I've forked the repo and will look at the code for this to see if I can't find anything. FYI, munin_passenger_status is returning actual numbers (it's config variables aren't quoted). munin_passenger_queue is returning 0's also, but I'm thinking that this actual data as the server is under a pretty light load.

*** Update ***
In an irb session I ran passenger-memory-stats and I do get a return value as expected from this program. However, the regex on line 37 of lib/munin/plugins/passenger_memory_stats.rb

stats =~ /RSS:\s*([\d\.]+)\s*MB\Z/m
memory = ($1.to_f * 1024 * 1024).to_i

This is looking for the string 'RSS: ' correct? This token doesn't appear in the output of passenger-memory-stats on FreeBSD. Here's the output I receive when running it from the command line:

---------- Apache processes ----------

PID PPID VMSize Resident Name

1691 1 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
1700 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
1701 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
1702 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
1703 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
1704 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start
43565 1 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90229 43565 76.5 MB 8.5 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90265 43565 76.5 MB 8.4 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90266 43565 76.5 MB 8.4 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90268 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90272 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90273 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90281 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90295 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90296 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90297 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90298 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90301 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90307 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT
90308 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT

-------- Nginx processes ---------

----- Passenger processes ------

PID VMSize Resident Name

1692 15.2 MB 3.1 MB PassengerWatchdog
1694 17.5 MB 3.7 MB PassengerHelperAgent
1696 29.4 MB 9.5 MB Passenger spawn server
1697 24.4 MB 6.0 MB PassengerLoggingAgent
84462 190.6 MB 131.7 MB Rack: /usr/local/docs/arc/current
88558 161.2 MB 107.1 MB Rack: /usr/local/docs/arc/current
90245 148.0 MB 98.4 MB Passenger ApplicationSpawner: /usr/local/docs/arc/current
90253 159.2 MB 105.3 MB Rack: /usr/local/docs/arc/current
94888 15.2 MB 3.2 MB PassengerWatchdog
94890 21.1 MB 7.5 MB PassengerHelperAgent
94892 29.4 MB 10.6 MB Passenger spawn server
94893 24.4 MB 6.3 MB PassengerLoggingAgent

[wakemaster39]

Yes RSS is correct, it must be an issue with FreeBSD and not reporting the RSS. On my linux boxes, you get an output like this.
-------- Apache processes ---------

PID PPID VMSize Private Name

2229 1 90.8 MB 0.2 MB /usr/sbin/apache2 -k start
2259 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2260 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2261 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2262 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2263 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2392 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2408 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start
2409 2229 90.8 MB 0.2 MB /usr/sbin/apache2 -k start

Processes: 9

Total private dirty RSS: 2.19 MB

That RSS is what it is looking for, though I am not sure why it isn't there on FreeBSD must have to do with memory management differences between the two kernels. I suppose it would be possible to do a look and add up all the VMSize memory, but I feel this is an issue with the passenger gem and not the munin plugin, though that is only my opinion.

[dogenpunk]

Checking the Passenger source, there's a call to #platform_provides_dirty_rss_information? that basically just checks to see if ruby_platform =~ /linux/. I'll look into seeing if there isn't a different way to determine this, but I'm guessing that I may be out of luck. Would you be interested in me updating the code to tally these numbers on non-linux platforms?

_Update_
I need to start checking the docs before I make these comments. From the Passenger User Guide:

Private dirty RSS reporting only works on Linux. Unfortunately other operating systems don’t provide facilities for determining
processes' private dirty RSS. On non-Linux systems, the Resident Set Size is reported instead.

So I'm guessing that these guys know what they're talking about. Would updating this plugin to report Resident Set Size on non-linux OSes be alright with you? If so, I'll close this and submit a pull request when I have it done.

Thanks for your help!

[barttenbrinke]

Wow, this issue actually resolved itself :)! Thanks Wakemaster39!
I'll add a note to the docs referring to the passenger user guide.
The rails 3 log format specification is allready in the docs, but as this is the second ticket about this I think i'll put env.log_format rails3 in as default. As this is probably what people expect.

As for the plugin update.. pull requests are allways welcome. It really depends on 1) the amount of code it will take and 2) if it is possible at all (given the Passenger docs comment).