Having a weird problem.
Using Nagios to monitor 2 servers, using check_all_procs, and every morning on both server, Nagios stops reporting the processes correctly.
Typically they are between 110-140, but every morning around 4am ET they shoot down to 2-3.
Reboot the servers, and all is well again until around 4am.
Nothing is running on the servers at that time to be causing an issue.
Anyone have any suggestions where to start looking?
Thanks,
David
check_all_procs start reporting wrong every morning around 4
-
- Posts: 7
- Joined: Sat Feb 01, 2020 9:44 pm
- Location: Oak Ridge, TN
-
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
Re: check_all_procs start reporting wrong every morning arou
What is in the logs?
/var/log/messages
/usr/local/nagios/var/nagios.log
You may need to enable debug logging on Nagios, try setting the debug level on and then restart Nagios.
Check the file /usr/local/nagios/var/nagios.debug
When you are finished this turns debugging off:
/var/log/messages
/usr/local/nagios/var/nagios.log
You may need to enable debug logging on Nagios, try setting the debug level on and then restart Nagios.
Code: Select all
sed -i 's/.*debug_level=.*/debug_level=-1/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart
When you are finished this turns debugging off:
Code: Select all
sed -i 's/.*debug_level=.*/debug_level=0/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 7
- Joined: Sat Feb 01, 2020 9:44 pm
- Location: Oak Ridge, TN
Re: check_all_procs start reporting wrong every morning arou
Only thing I am seeing close to that time is the log rotation on the server.
But don't see how that would be affection Nagios.
But don't see how that would be affection Nagios.
-
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
Re: check_all_procs start reporting wrong every morning arou
Did you enable debug logging ?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 7
- Joined: Sat Feb 01, 2020 9:44 pm
- Location: Oak Ridge, TN
Re: check_all_procs start reporting wrong every morning arou
I did enable it, and it failed on 2 of the 3 servers being monitored.
All are running CentOS 7 and CWP.
Can I PM you the debug log to look at?
All are running CentOS 7 and CWP.
Can I PM you the debug log to look at?
-
- Posts: 5324
- Joined: Wed Aug 22, 2018 4:39 pm
- Location: saint paul
Re: check_all_procs start reporting wrong every morning arou
Hello David,
Certainly. Please send it to me in PM and I can share this with Box293. Thanks.Can I PM you the debug log to look at?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Be sure to check out our Knowledgebase for helpful articles and solutions!
-
- Posts: 7
- Joined: Sat Feb 01, 2020 9:44 pm
- Location: Oak Ridge, TN
Re: check_all_procs start reporting wrong every morning arou
I don't think this is a Nagios XI problem, since the 3rd server which happens to be a Non Pro CWP server is working correctly with stats.
Attached is the debug file, if someone can look just to double check it.
ATL3 and ATL6 have the problem, and have to be rebooted to start reporting correctly again, but EWR3 is OK.
Thanks,
David
Attached is the debug file, if someone can look just to double check it.
ATL3 and ATL6 have the problem, and have to be rebooted to start reporting correctly again, but EWR3 is OK.
Thanks,
David
You do not have the required permissions to view the files attached to this post.
-
- Posts: 7
- Joined: Sat Feb 01, 2020 9:44 pm
- Location: Oak Ridge, TN
Re: check_all_procs start reporting wrong every morning arou
Here is the graph from Nagios.
Weird.
All is well after a manually reboot of the server.
Weird.
All is well after a manually reboot of the server.
You do not have the required permissions to view the files attached to this post.
-
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
Re: check_all_procs start reporting wrong every morning arou
This completely seems like an issue on your servers and nothing to do with Nagios.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.