/usr/local/nagios/var/service-perfdata 5.6G in size

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
hbouma
Posts: 483
Joined: Tue Feb 27, 2018 9:31 am

/usr/local/nagios/var/service-perfdata 5.6G in size

Post by hbouma »

We are running Nagios XI 5.8.3 on RHEL 7 VM's.

On the server, we are seeing a 5.6G /usr/local/nagios/var/service-perfdata. This is our DEV server with only 6 hosts and 96 services for testing.

On our Production servers, the file is only around 11K in size, even though they have 700+ Hosts and nearly 7,000 services.

Any ideas as to why this file is growing so large?
Top
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by benjaminsmith »

Hi Henry,

That indicates the system is not processing the performance data. Have checked the performance graphs, are they updating?

Have you made any upgrades lately or system changes?

Try to restart npcd and nagios, and let me know if that helps. It maybe necessary to clean out that file.

Code: Select all

systemctl restart npcd
systemctl restart nagios
If that doesn't work, send us the system profile, and we'll check the settings and the commands.

Best Regards,
Benjamin
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
hbouma
Posts: 483
Joined: Tue Feb 27, 2018 9:31 am

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by hbouma »

We recently upgraded to Nagios 5.8.3 and updated to NDO3. You are correct that the graphs are not updating. Can you work with us to get NDO3 working?

The perdataproc.log file is showing the following:

DONE. Processed 0 files.

DONE. Processed 0 files.
Outbound data DISABLED Mon, 03 May 2021 08:47:01 -0400

DONE. Processed 0 files.
Outbound data DISABLED Mon, 03 May 2021 08:48:01 -0400

DONE. Processed 0 files.
Outbound data DISABLED Mon, 03 May 2021 08:49:01 -0400

DONE. Processed 0 files.
Outbound data DISABLED Mon, 03 May 2021 08:50:02 -0400
Outbound data DISABLED Mon, 03 May 2021 08:51:01 -0400
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by ssax »

Please follow this guide to troubleshoot:

https://support.nagios.com/kb/article.php?id=9

Please PM me a copy of your profile, you can download it from Admin > System Profile by clicking the Download Profile button so I can analyze your setup to determine what is wrong.
Top
hbouma
Posts: 483
Joined: Tue Feb 27, 2018 9:31 am

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by hbouma »

The profile will be sent shortly in a PM.

I see nothing in the output from the commands in that document that would indicate a problem. I have attached them to this message for you to review.

Code: Select all

ls /usr/local/nagios/var/spool/perfdata/ | wc -l
0

ls /usr/local/nagios/var/spool/xidpe/ | wc -l
0

After increasing the Performance Data log level, I ran the tail command listed and got the following from 2019 and 2020:

Code: Select all

tail -f /usr/local/nagios/var/perfdata.log
2019-09-12 20:18:43 [30624] [0] *** TIMEOUT: Please check your npcd.cfg
2019-09-12 20:18:43 [30624] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1568333895.perfdata.host-PID-30624 deleted
2019-09-12 20:18:43 [30624] [0] *** Timeout while processing Host: "" Service: ""
2019-09-12 20:18:43 [30624] [0] *** process_perfdata.pl terminated on signal ALRM
2020-09-05 03:58:00 [24051] [0] *** TIMEOUT: Timeout after 5 secs. ***
2020-09-05 03:58:01 [24051] [0] *** TIMEOUT: Deleting current file to avoid NPCD loops
2020-09-05 03:58:01 [24051] [0] *** TIMEOUT: Please check your npcd.cfg
2020-09-05 03:58:01 [24051] [0] *** TIMEOUT: /usr/local/nagios/var/spool/perfdata//1599292649.perfdata.service-PID-24051 deleted
2020-09-05 03:58:01 [24051] [0] *** Timeout while processing Host: "localhost" Service: "Total_Processes"
2020-09-05 03:58:01 [24051] [0] *** process_perfdata.pl terminated on signal ALRM
I performed the changes to the npcd logging and received the following output:

Code: Select all

[05-03-2021 14:50:41] NPCD: Caught Termination Signal - Hasta la vista... baby
[05-03-2021 14:50:41] NPCD: npcd Daemon (0.4.14) started with PID=29933
[05-03-2021 14:50:41] NPCD: Please have a look at 'npcd -V' to get license information
[05-03-2021 14:50:41] NPCD: HINT: load_threshold is enabled - ('10.000000')
[05-03-2021 14:50:41] NPCD: Found 2 files in /usr/local/nagios/var/spool/perfdata/
[05-03-2021 14:50:41] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:50:41] NPCD: ThreadCounter 0/5 File is .
[05-03-2021 14:50:41] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:50:41] NPCD: ThreadCounter 0/5 File is ..
[05-03-2021 14:50:41] NPCD: No more files to process... waiting for 15 seconds
[05-03-2021 14:50:56] NPCD: Found 2 files in /usr/local/nagios/var/spool/perfdata/
[05-03-2021 14:50:56] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:50:56] NPCD: ThreadCounter 0/5 File is .
[05-03-2021 14:50:56] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:50:56] NPCD: ThreadCounter 0/5 File is ..
[05-03-2021 14:50:56] NPCD: No more files to process... waiting for 15 seconds
[05-03-2021 14:51:11] NPCD: Found 2 files in /usr/local/nagios/var/spool/perfdata/
[05-03-2021 14:51:11] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:11] NPCD: ThreadCounter 0/5 File is .
[05-03-2021 14:51:11] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:11] NPCD: ThreadCounter 0/5 File is ..
[05-03-2021 14:51:11] NPCD: No more files to process... waiting for 15 seconds
[05-03-2021 14:51:26] NPCD: Found 2 files in /usr/local/nagios/var/spool/perfdata/
[05-03-2021 14:51:26] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:26] NPCD: ThreadCounter 0/5 File is .
[05-03-2021 14:51:26] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:26] NPCD: ThreadCounter 0/5 File is ..
[05-03-2021 14:51:26] NPCD: No more files to process... waiting for 15 seconds
[05-03-2021 14:51:41] NPCD: Found 2 files in /usr/local/nagios/var/spool/perfdata/
[05-03-2021 14:51:41] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:41] NPCD: ThreadCounter 0/5 File is .
[05-03-2021 14:51:41] NPCD: DEBUG: load 0.000000/10.000000
[05-03-2021 14:51:41] NPCD: ThreadCounter 0/5 File is ..
[05-03-2021 14:51:41] NPCD: No more files to process... waiting for 15 seconds
The user account is not expired:

Code: Select all

chage -l nagios
Last password change                                    : Aug 31, 2018
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 14
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by ssax »

The issue is that you have a ramdisk half-implemented.

Please follow page 3+ of the guide below to update all the files again to use the ramdisk:
- You can try the auto installation but it likely won't allow you since it's half-implemented

https://assets.nagios.com/downloads/nag ... giosXI.pdf

Additionally, you'll need to truncate those files as well as that's too big to process:

Code: Select all

truncate -s 0 /usr/local/nagios/var/host-perfdata
truncate -s 0 /usr/local/nagios/var/service-perfdata
Top
hbouma
Posts: 483
Joined: Tue Feb 27, 2018 9:31 am

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by hbouma »

We have made the changes recommended, but are not seeing any performance data at this time.
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by ssax »

Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

Code: Select all

https://support.nagios.com/tickets/
Attach a fresh copy of your profile to the ticket as well so I can look at the configs/logs.
EDIT: Just saw the PM

Thank you!
Top
hbouma
Posts: 483
Joined: Tue Feb 27, 2018 9:31 am

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by hbouma »

Thank you. Fixing the RAMDISK fixed our issue.
Top
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: /usr/local/nagios/var/service-perfdata 5.6G in size

Post by scottwilkerson »

hbouma wrote:Thank you. Fixing the RAMDISK fixed our issue.
Locking thread
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Top

Return to “Nagios XI”