Blank notifications page, npcd running wild, load average hi

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
jon.bilderback
Posts: 52
Joined: Tue Aug 10, 2010 9:16 am

Blank notifications page, npcd running wild, load average hi

Post by jon.bilderback »

Under the home tab -> incident management -> Notifications, the page does not come up, just stays blank.
In the /var/log/httpd/error_log file, i get the following:

PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 19887545 bytes) in /usr/local/nagiosxi/html/includes/components/nagioscore/coreuiproxy.inc.php on line 154, referer: http://nagios.lamar.edu/nagiosxi/

I have edited the /etc/php.ini file to change memory_limit from 16M to 128M, but still have the same issue.

The npcd service is using somewhere in the neighborhood of 80 %CPU when I look at it in "top", but I do not know if that is related.
My load average is 4.52, 4.11, 4.19.

VMWare install of server, updated to 2011r1.8
agriffin
Posts: 876
Joined: Mon May 09, 2011 9:36 am

Re: Blank notifications page, npcd running wild, load averag

Post by agriffin »

There are a few things about this that seem strange to me. For example, npcd normally turns itself off when the load is too high. But I think it will be easier to take things one at a time. Try disabling npcd:

Code: Select all

service npcd stop
Afterwards, let me know if that fixes anything or if there are still problems. Note that performance graphs will stop updating; this is to be expected when npcd is not running. If everything else works we'll move on to re-enabling npcd and trying to reduce the load it's using.
jon.bilderback
Posts: 52
Joined: Tue Aug 10, 2010 9:16 am

Re: Blank notifications page, npcd running wild, load averag

Post by jon.bilderback »

I stopped the npcd service, and within a few minutes : load average: 1.32, 2.54, 3.50
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: Blank notifications page, npcd running wild, load averag

Post by mguthrie »

If your environment is large I would recommend increasing your php memory limit as high as 512.

npcd can eat up a fair amount of CPU with a high check volume, or if the process has been stopped for a while. I would check the /usr/local/nagios/var/npcd.log and make sure it's processing files ok and that there aren't permissions problems. If there are permissions issues with a large number of hosts and services it's possible that npcd would eat up a fair amount of CPU.
http://assets.nagios.com/downloads/nagi ... p#fixperms

How large is your environment (how many hosts + services?)
jon.bilderback
Posts: 52
Joined: Tue Aug 10, 2010 9:16 am

Re: Blank notifications page, npcd running wild, load averag

Post by jon.bilderback »

I changed the memory_limit in /etc/php.ini to 512M, but I'm still getting the same results:
PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 20111998 bytes) in /usr/local/nagiosxi/html/includes/components/nagioscore/coreuiproxy.inc.php on line 154, referer: http://nagios.lamar.edu/nagiosxi/index.php?

316 Hosts, 1853 services.

I ran the permission script, and restarted npcd, but still see the immediate spike in cpu load.
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: Blank notifications page, npcd running wild, load averag

Post by mguthrie »

Ok, I think I have an idea as to what's going on for the memory error. Can you access your nagios service with the http://<youraddress>/nagios and pull up the notifications page. About how many notifications would you say you have on that page? (rough ballpark)

For PNP, check your logging settings in the following files.

/usr/local/nagios/etc/pnp/npcd.cfg
/usr/local/nagios/etc/pnp/process_perfdata.cfg

and try setting the logging to 0 if it's not already. Increased logging can increase CPU usage.

If that doesn't resolve the issue, you can turn logging back on and then watch the following log files for any clues:

/usr/local/nagios/var/perfdata.log
/usr/local/nagios/var/npcd.log
jon.bilderback
Posts: 52
Joined: Tue Aug 10, 2010 9:16 am

Re: Blank notifications page, npcd running wild, load averag

Post by jon.bilderback »

from npcd.log:
[11-03-2011 12:37:20] NPCD: ERROR: Executed command exits with return code '7'
[11-03-2011 12:37:20] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//service-perfdata.1320341816'

I have set the logging to "0", and it looks like the cpu load is staying pretty low, and the performance data is showing up again in graphs.

For some reason, my password is not working on the http://<youraddress>/nagios page. Is this the same as the core config nagiosadmin password? If so, it isn't working. If not, how do I change that password?

Thanks
User avatar
lmiltchev
Former Nagios Staff
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Blank notifications page, npcd running wild, load averag

Post by lmiltchev »

Your login pasword (Core & XI) and Nagios Core Config Manager password don't have to be the same.

If you are having login issues, you may try resetting your password by running in terminal:

Code: Select all

# cd /usr/local/nagiosxi/scripts
# ./reset_nagiosadmin_password.php --password=<newpassword>
where <newpassword> is your new password.

Then you can try to log in again:

In XI: http://<your_IP_address>/nagiosxi
In Core: http://<your_IP_address>/nagios

Hope this helps.
Be sure to check out our Knowledgebase for helpful articles and solutions!
jon.bilderback
Posts: 52
Joined: Tue Aug 10, 2010 9:16 am

Re: Blank notifications page, npcd running wild, load averag

Post by jon.bilderback »

For some reason, even after resetting the nagiosadmin password in the terminal, I am still not able to get to that url: http://<youraddress>/nagios

Is there another place or way to change that password?
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: Blank notifications page, npcd running wild, load averag

Post by mguthrie »

Yeah, the /nagios URL requires an apache authentication. You can set this through the Admin->Reset Security Credentials and manually set if for the nagiosadmin user. If you want to add additional users you can do so with the following command:

Code: Select all

cd /usr/local/nagiosxi/etc
htpasswd htpasswd.users <username>
You'll be prompted to update the password.