graphs stopped working after upgrade and update to 2012R1.5b

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
marquetteu
Posts: 47
Joined: Tue Nov 13, 2012 12:08 pm

graphs stopped working after upgrade and update to 2012R1.5b

Post by marquetteu »

after doing a yum update and then an update to 2012R1.5b all my graphs no longer show any data saying nan for all values.

Please advise
Top
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: graphs stopped working after upgrade and update to 2012R

Post by abrist »

Could you post the following log files in a code wrap?

Code: Select all

tail -50 /usr/local/nagios/var/perfdata.log
tail -50 /usr/local/nagios/var/npcd.log
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Top
marquetteu
Posts: 47
Joined: Tue Nov 13, 2012 12:08 pm

Re: graphs stopped working after upgrade and update to 2012R

Post by marquetteu »

Code: Select all

$ tail -50 /usr/local/nagios/var/perfdata.log
2013-01-30 03:14:53 [25682] [0] *** Timeout while processing Host: "its-mjdev3" Service: "CPU"
2013-01-30 03:14:53 [25682] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service-PID-28455 deleted
2013-01-30 03:19:31 [28455] [0] *** Timeout while processing Host: "its-gporadev1" Service: "CPU"
2013-01-30 03:19:31 [28455] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service-PID-3883 deleted
2013-01-30 03:34:19 [3883] [0] *** Timeout while processing Host: "its-psoradev2" Service: "CPU"
2013-01-30 03:34:19 [3883] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host-PID-27945 deleted
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service-PID-27946 deleted
2013-01-30 04:16:24 [27945] [0] *** Timeout while processing Host: "vs-empsnd" Service: "_HOST_"
2013-01-30 04:16:24 [27946] [0] *** Timeout while processing Host: "vs-oemstg" Service: "DiskIO"
2013-01-30 04:16:24 [27945] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27946] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host-PID-4713:No such file or directory
2013-01-30 04:33:32 [4713] [0] *** Timeout while processing Host: "" Service: ""
2013-01-30 04:33:32 [4713] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service-PID-4714:No such file or directory
2013-01-30 04:33:32 [4714] [0] *** Timeout while processing Host: "its-oradev1" Service: "Paging"
2013-01-30 04:33:32 [4714] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service-PID-29495 deleted
2013-01-30 12:00:08 [29495] [0] *** Timeout while processing Host: "its-mjprod1" Service: "CPU"
2013-01-30 12:00:08 [29495] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service-PID-5415 deleted
2013-01-30 21:50:14 [5415] [0] *** Timeout while processing Host: "its-mjprod2" Service: "CPU"
2013-01-30 21:50:14 [5415] [0] *** process_perfdata.pl terminated on signal ALRM

Code: Select all

$ tail -50 /usr/local/nagios/var/npcd.log
[01-27-2013 01:06:26] NPCD: WARN: MAX load reached: load 10.300000/10.000000 at i=0[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270369.perfdata.host'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270335.perfdata.host'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270320.perfdata.service'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270335.perfdata.service'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270320.perfdata.host'
[01-27-2013 01:07:08] NPCD: WARN: MAX load reached: load 12.200000/10.000000 at i=7[01-27-2013 01:07:23] NPCD: WARN: MAX load reached: load 12.880000/10.000000 at i=7[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270370.perfdata.service'
[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270369.perfdata.service'
[01-27-2013 01:09:13] NPCD: WARN: MAX load reached: load 25.890000/10.000000 at i=0[01-27-2013 01:09:28] NPCD: WARN: MAX load reached: load 20.390000/10.000000 at i=1[01-27-2013 01:09:43] NPCD: WARN: MAX load reached: load 16.030000/10.000000 at i=1[01-27-2013 01:09:58] NPCD: WARN: MAX load reached: load 14.580000/10.000000 at i=1[01-27-2013 01:11:08] NPCD: WARN: MAX load reached: load 32.400000/10.000000 at i=1[01-27-2013 01:11:23] NPCD: WARN: MAX load reached: load 27.600000/10.000000 at i=1[01-27-2013 01:11:38] NPCD: WARN: MAX load reached: load 21.690000/10.000000 at i=1[01-27-2013 01:11:53] NPCD: WARN: MAX load reached: load 17.810000/10.000000 at i=1[01-27-2013 01:12:46] NPCD: WARN: MAX load reached: load 23.460000/10.000000 at i=1[01-27-2013 01:13:01] NPCD: WARN: MAX load reached: load 18.270000/10.000000 at i=1[01-27-2013 01:13:16] NPCD: WARN: MAX load reached: load 14.220000/10.000000 at i=1[01-27-2013 01:13:31] NPCD: WARN: MAX load reached: load 11.070000/10.000000 at i=1[01-27-2013 01:14:04] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:14:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270386.perfdata.service'
[01-29-2013 17:00:55] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-29-2013 17:02:46] NPCD: npcd Daemon (0.4.14) started with PID=5024
[01-29-2013 17:02:46] NPCD: Please have a look at 'npcd -V' to get license information
[01-29-2013 17:02:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-30-2013 02:59:25] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 02:59:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536346.perfdata.service'
[01-30-2013 03:00:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:00:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536421.perfdata.service'
[01-30-2013 03:04:15] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:04:15] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536631.perfdata.host'
[01-30-2013 03:10:11] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:10:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536991.perfdata.service'
[01-30-2013 03:14:53] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:14:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537276.perfdata.service'
[01-30-2013 03:19:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:19:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service'
[01-30-2013 03:34:19] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:34:19] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service'
[01-30-2013 12:00:08] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 12:00:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service'
[01-30-2013 21:50:14] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 21:50:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service'
[01-31-2013 08:51:58] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-04-2013 11:04:07] NPCD: npcd Daemon (0.4.14) started with PID=5082
[02-04-2013 11:04:07] NPCD: Please have a look at 'npcd -V' to get license information
[02-04-2013 11:04:07] NPCD: HINT: load_threshold is enabled - ('10.000000')
Top
slansing
Posts: 7698
Joined: Mon Apr 23, 2012 4:28 pm
Location: Travelling through time and space...

Re: graphs stopped working after upgrade and update to 2012R

Post by slansing »

Can you run the following and then tail the logs again?:

Edit /usr/local/nagios/etc/pnp/npcd.cfg:

Change the load threshold:

Code: Select all

load_threshold = 30.0

Code: Select all

service npcd restart
Top
marquetteu
Posts: 47
Joined: Tue Nov 13, 2012 12:08 pm

Re: graphs stopped working after upgrade and update to 2012R

Post by marquetteu »

Code: Select all

$ tail -50 /usr/local/nagios/var/perfdata.log
2013-01-30 03:14:53 [25682] [0] *** Timeout while processing Host: "its-mjdev3" Service: "CPU"
2013-01-30 03:14:53 [25682] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service-PID-28455 deleted
2013-01-30 03:19:31 [28455] [0] *** Timeout while processing Host: "its-gporadev1" Service: "CPU"
2013-01-30 03:19:31 [28455] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service-PID-3883 deleted
2013-01-30 03:34:19 [3883] [0] *** Timeout while processing Host: "its-psoradev2" Service: "CPU"
2013-01-30 03:34:19 [3883] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host-PID-27945 deleted
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service-PID-27946 deleted
2013-01-30 04:16:24 [27945] [0] *** Timeout while processing Host: "vs-empsnd" Service: "_HOST_"
2013-01-30 04:16:24 [27946] [0] *** Timeout while processing Host: "vs-oemstg" Service: "DiskIO"
2013-01-30 04:16:24 [27945] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27946] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host-PID-4713:No such file or directory
2013-01-30 04:33:32 [4713] [0] *** Timeout while processing Host: "" Service: ""
2013-01-30 04:33:32 [4713] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service-PID-4714:No such file or directory
2013-01-30 04:33:32 [4714] [0] *** Timeout while processing Host: "its-oradev1" Service: "Paging"
2013-01-30 04:33:32 [4714] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service-PID-29495 deleted
2013-01-30 12:00:08 [29495] [0] *** Timeout while processing Host: "its-mjprod1" Service: "CPU"
2013-01-30 12:00:08 [29495] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service-PID-5415 deleted
2013-01-30 21:50:14 [5415] [0] *** Timeout while processing Host: "its-mjprod2" Service: "CPU"
2013-01-30 21:50:14 [5415] [0] *** process_perfdata.pl terminated on signal ALRM

Code: Select all

$ tail -50 /usr/local/nagios/var/npcd.log
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270320.perfdata.service'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270335.perfdata.service'
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270320.perfdata.host'
[01-27-2013 01:07:08] NPCD: WARN: MAX load reached: load 12.200000/10.000000 at i=7[01-27-2013 01:07:23] NPCD: WARN: MAX load reached: load 12.880000/10.000000 at i=7[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270370.perfdata.service'
[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270369.perfdata.service'
[01-27-2013 01:09:13] NPCD: WARN: MAX load reached: load 25.890000/10.000000 at i=0[01-27-2013 01:09:28] NPCD: WARN: MAX load reached: load 20.390000/10.000000 at i=1[01-27-2013 01:09:43] NPCD: WARN: MAX load reached: load 16.030000/10.000000 at i=1[01-27-2013 01:09:58] NPCD: WARN: MAX load reached: load 14.580000/10.000000 at i=1[01-27-2013 01:11:08] NPCD: WARN: MAX load reached: load 32.400000/10.000000 at i=1[01-27-2013 01:11:23] NPCD: WARN: MAX load reached: load 27.600000/10.000000 at i=1[01-27-2013 01:11:38] NPCD: WARN: MAX load reached: load 21.690000/10.000000 at i=1[01-27-2013 01:11:53] NPCD: WARN: MAX load reached: load 17.810000/10.000000 at i=1[01-27-2013 01:12:46] NPCD: WARN: MAX load reached: load 23.460000/10.000000 at i=1[01-27-2013 01:13:01] NPCD: WARN: MAX load reached: load 18.270000/10.000000 at i=1[01-27-2013 01:13:16] NPCD: WARN: MAX load reached: load 14.220000/10.000000 at i=1[01-27-2013 01:13:31] NPCD: WARN: MAX load reached: load 11.070000/10.000000 at i=1[01-27-2013 01:14:04] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:14:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270386.perfdata.service'
[01-29-2013 17:00:55] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-29-2013 17:02:46] NPCD: npcd Daemon (0.4.14) started with PID=5024
[01-29-2013 17:02:46] NPCD: Please have a look at 'npcd -V' to get license information
[01-29-2013 17:02:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-30-2013 02:59:25] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 02:59:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536346.perfdata.service'
[01-30-2013 03:00:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:00:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536421.perfdata.service'
[01-30-2013 03:04:15] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:04:15] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536631.perfdata.host'
[01-30-2013 03:10:11] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:10:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536991.perfdata.service'
[01-30-2013 03:14:53] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:14:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537276.perfdata.service'
[01-30-2013 03:19:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:19:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service'
[01-30-2013 03:34:19] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:34:19] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service'
[01-30-2013 12:00:08] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 12:00:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service'
[01-30-2013 21:50:14] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 21:50:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service'
[01-31-2013 08:51:58] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-04-2013 11:04:07] NPCD: npcd Daemon (0.4.14) started with PID=5082
[02-04-2013 11:04:07] NPCD: Please have a look at 'npcd -V' to get license information
[02-04-2013 11:04:07] NPCD: HINT: load_threshold is enabled - ('10.000000')
[02-04-2013 14:47:54] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-04-2013 14:47:54] NPCD: npcd Daemon (0.4.14) started with PID=22216
[02-04-2013 14:47:54] NPCD: Please have a look at 'npcd -V' to get license information
[02-04-2013 14:47:54] NPCD: HINT: load_threshold is enabled - ('30.000000')
Top
slansing
Posts: 7698
Joined: Mon Apr 23, 2012 4:28 pm
Location: Travelling through time and space...

Re: graphs stopped working after upgrade and update to 2012R

Post by slansing »

Can you post your system load?

Report the output of the following:

Code: Select all

service npcd stop

Code: Select all

killall npcd
At this point give the system a minute to spool a bit and check current system load:

Code: Select all

top
Then restart npcd and check the logs again, we may have to increase the threshold more:

Code: Select all

service npcd start
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: graphs stopped working after upgrade and update to 2012R

Post by scottwilkerson »

Can you run and report back

Code: Select all

ls -l /usr/local/nagios/var/spool/perfdata|wc -l
Also lets increase the TIMEOUT in /usr/local/nagios/etc/pnp/process_perfdata.cfg to 15
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top
marquetteu
Posts: 47
Joined: Tue Nov 13, 2012 12:08 pm

Re: graphs stopped working after upgrade and update to 2012R

Post by marquetteu »

Code: Select all

 tail -50 /usr/local/nagios/var/npcd.log
[01-27-2013 01:07:08] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:07:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270320.perfdata.host'
[01-27-2013 01:07:08] NPCD: WARN: MAX load reached: load 12.200000/10.000000 at i=7[01-27-2013 01:07:23] NPCD: WARN: MAX load reached: load 12.880000/10.000000 at i=7[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270370.perfdata.service'
[01-27-2013 01:08:58] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:08:58] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270369.perfdata.service'
[01-27-2013 01:09:13] NPCD: WARN: MAX load reached: load 25.890000/10.000000 at i=0[01-27-2013 01:09:28] NPCD: WARN: MAX load reached: load 20.390000/10.000000 at i=1[01-27-2013 01:09:43] NPCD: WARN: MAX load reached: load 16.030000/10.000000 at i=1[01-27-2013 01:09:58] NPCD: WARN: MAX load reached: load 14.580000/10.000000 at i=1[01-27-2013 01:11:08] NPCD: WARN: MAX load reached: load 32.400000/10.000000 at i=1[01-27-2013 01:11:23] NPCD: WARN: MAX load reached: load 27.600000/10.000000 at i=1[01-27-2013 01:11:38] NPCD: WARN: MAX load reached: load 21.690000/10.000000 at i=1[01-27-2013 01:11:53] NPCD: WARN: MAX load reached: load 17.810000/10.000000 at i=1[01-27-2013 01:12:46] NPCD: WARN: MAX load reached: load 23.460000/10.000000 at i=1[01-27-2013 01:13:01] NPCD: WARN: MAX load reached: load 18.270000/10.000000 at i=1[01-27-2013 01:13:16] NPCD: WARN: MAX load reached: load 14.220000/10.000000 at i=1[01-27-2013 01:13:31] NPCD: WARN: MAX load reached: load 11.070000/10.000000 at i=1[01-27-2013 01:14:04] NPCD: ERROR: Executed command exits with return code '7'
[01-27-2013 01:14:04] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359270386.perfdata.service'
[01-29-2013 17:00:55] NPCD: Caught Termination Signal - Hasta la vista... baby
[01-29-2013 17:02:46] NPCD: npcd Daemon (0.4.14) started with PID=5024
[01-29-2013 17:02:46] NPCD: Please have a look at 'npcd -V' to get license information
[01-29-2013 17:02:46] NPCD: HINT: load_threshold is enabled - ('10.000000')
[01-30-2013 02:59:25] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 02:59:25] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536346.perfdata.service'
[01-30-2013 03:00:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:00:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536421.perfdata.service'
[01-30-2013 03:04:15] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:04:15] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536631.perfdata.host'
[01-30-2013 03:10:11] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:10:11] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359536991.perfdata.service'
[01-30-2013 03:14:53] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:14:53] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537276.perfdata.service'
[01-30-2013 03:19:31] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:19:31] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service'
[01-30-2013 03:34:19] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 03:34:19] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service'
[01-30-2013 04:16:24] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host'
[01-30-2013 04:33:32] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 04:33:32] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service'
[01-30-2013 12:00:08] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 12:00:08] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service'
[01-30-2013 21:50:14] NPCD: ERROR: Executed command exits with return code '7'
[01-30-2013 21:50:14] NPCD: ERROR: Command line was '/usr/local/nagios/libexec/process_perfdata.pl -n -b /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service'
[01-31-2013 08:51:58] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-04-2013 11:04:07] NPCD: npcd Daemon (0.4.14) started with PID=5082
[02-04-2013 11:04:07] NPCD: Please have a look at 'npcd -V' to get license information
[02-04-2013 11:04:07] NPCD: HINT: load_threshold is enabled - ('10.000000')
[02-04-2013 14:47:54] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-04-2013 14:47:54] NPCD: npcd Daemon (0.4.14) started with PID=22216
[02-04-2013 14:47:54] NPCD: Please have a look at 'npcd -V' to get license information
[02-04-2013 14:47:54] NPCD: HINT: load_threshold is enabled - ('30.000000')
[02-05-2013 08:57:31] NPCD: Caught Termination Signal - Hasta la vista... baby
[02-05-2013 09:00:14] NPCD: npcd Daemon (0.4.14) started with PID=18577
[02-05-2013 09:00:14] NPCD: Please have a look at 'npcd -V' to get license information
[02-05-2013 09:00:14] NPCD: HINT: load_threshold is enabled - ('30.000000')

Code: Select all

$ tail -50 /usr/local/nagios/var/perfdata.log
2013-01-30 03:14:53 [25682] [0] *** Timeout while processing Host: "its-mjdev3" Service: "CPU"
2013-01-30 03:14:53 [25682] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:19:31 [28455] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359537561.perfdata.service-PID-28455 deleted
2013-01-30 03:19:31 [28455] [0] *** Timeout while processing Host: "its-gporadev1" Service: "CPU"
2013-01-30 03:19:31 [28455] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 03:34:19 [3883] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359538446.perfdata.service-PID-3883 deleted
2013-01-30 03:34:19 [3883] [0] *** Timeout while processing Host: "its-psoradev2" Service: "CPU"
2013-01-30 03:34:19 [3883] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:16:24 [27945] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.host-PID-27945 deleted
2013-01-30 04:16:24 [27946] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359540966.perfdata.service-PID-27946 deleted
2013-01-30 04:16:24 [27945] [0] *** Timeout while processing Host: "vs-empsnd" Service: "_HOST_"
2013-01-30 04:16:24 [27946] [0] *** Timeout while processing Host: "vs-oemstg" Service: "DiskIO"
2013-01-30 04:16:24 [27945] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:16:24 [27946] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4713] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.host-PID-4713:No such file or directory
2013-01-30 04:33:32 [4713] [0] *** Timeout while processing Host: "" Service: ""
2013-01-30 04:33:32 [4713] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 04:33:32 [4714] [0] *** TIMEOUT: Could not delete /usr/local/nagios/var/spool/perfdata//1359542001.perfdata.service-PID-4714:No such file or directory
2013-01-30 04:33:32 [4714] [0] *** Timeout while processing Host: "its-oradev1" Service: "Paging"
2013-01-30 04:33:32 [4714] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 12:00:08 [29495] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359568791.perfdata.service-PID-29495 deleted
2013-01-30 12:00:08 [29495] [0] *** Timeout while processing Host: "its-mjprod1" Service: "CPU"
2013-01-30 12:00:08 [29495] [0] *** process_perfdata.pl terminated on signal ALRM
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Timeout after 5 secs. ***
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: Please check your npcd.cfg
2013-01-30 21:50:14 [5415] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1359604191.perfdata.service-PID-5415 deleted
2013-01-30 21:50:14 [5415] [0] *** Timeout while processing Host: "its-mjprod2" Service: "CPU"
2013-01-30 21:50:14 [5415] [0] *** process_perfdata.pl terminated on signal ALRM

Code: Select all

$ ls -l /usr/local/nagios/var/spool/perfdata|wc -l
3
I upped the timeout -- do which services do i need to bounce?

thanks!
Adam
Top
slansing
Posts: 7698
Joined: Mon Apr 23, 2012 4:28 pm
Location: Travelling through time and space...

Re: graphs stopped working after upgrade and update to 2012R

Post by slansing »

Did you run these log tails after you changed the timeout? It looks like they still have the old timeout rate, after changing it what do the logs show now?
Top
marquetteu
Posts: 47
Joined: Tue Nov 13, 2012 12:08 pm

Re: graphs stopped working after upgrade and update to 2012R

Post by marquetteu »

slansing wrote:Did you run these log tails after you changed the timeout? It looks like they still have the old timeout rate, after changing it what do the logs show now?
yes this was post timeout change. I just looked at the logs and nothing has been added since i restarted npcd
Top

Return to “Nagios XI”