Slow Nagios server
-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: Slow Nagios server
Are you using MySQL or MariaDB for your database server?
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
centricrm
- Posts: 29
- Joined: Tue Jun 05, 2018 3:58 am
Re: Slow Nagios server
We are you using MariaDB for our database server.
-
ssax
- Dreams In Code
- Posts: 7682
- Joined: Wed Feb 11, 2015 12:54 pm
Re: Slow Nagios server
Please go to Admin > System Profile and click the View System Info button and PM me the full output from that page, it's missing from your profile.
I would upgrade to XI 5.8.3 and see if that helps.
What is the output of these commands?
I would upgrade to XI 5.8.3 and see if that helps.
What is the output of these commands?
Code: Select all
mysql -uroot -pnagiosxi -e 'show full processlist;'
mysql -uroot -pnagiosxi nagios -e 'select count(*) from nagios_objects;'
cat /etc/my.cnf
cat /etc/my.cnf.d/*
ulimit -a
su -s /bin/bash -c 'ulimit -a' nagios
su -s /bin/bash -c 'ulimit -a' mysql
-
centricrm
- Posts: 29
- Joined: Tue Jun 05, 2018 3:58 am
Re: Slow Nagios server
Hi,
Two reports are sent in PM's.
Are there any changes in XI 5.8.3 that are related this issue?
wkr,
Humphrey
Two reports are sent in PM's.
Are there any changes in XI 5.8.3 that are related this issue?
wkr,
Humphrey
-
ssax
- Dreams In Code
- Posts: 7682
- Joined: Wed Feb 11, 2015 12:54 pm
Re: Slow Nagios server
Yes, it has an updated version of NDO3 that has some bugs fixed.
While likely unrelated, edit this file:
Change this:
To this:
Given the size of your system you should start looking at adding an additional XI server to reduce the total number of checks on the server:
An offloaded DB would not be recommended given that size of system.
Generally at 10K total combined host/service checks we recommend that you setup a RAMDisk (which you've done), and at around 20K we recommend you start looking at adding an additional XI server because they can only process so much. Now this may come sooner or later than 20K depending on what type of checks you are running, how much resources they use, your hardware speed, and what you're doing to mitigate the impact.
You should run this check profiler script and see what long running checks you have and determine what some of your long running checks are, they consume resources the whole time they are running so reducing those helps a lot:
https://exchange.nagios.org/directory/P ... me/details
Please read through this doc as well, with the number of checks you are running I would leave the DB local though at this point in time because of the large amount of total checks you have, it requires a lot of throughput to the DB (recommended enabling jumbo_frames):
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
You can only do so much on a single server, you'll need to do what you can to mitigate the impact but you should start looking at adding another XI server soon if you continue to experience load/performance issues after doing the mitigation.
Let me know if you have any questions or if I can clarify anything.
I did have a customer say he had better performance with the old NDO2DB, if you'd like to try that to see if it helps (I would):
Run these commands as root:
If you have an offloaded database you will need to edit your /usr/local/nagios/etc/ndo2db.cfg file and update these before running the next command to start it up:
- You can get the info from your /usr/local/nagios/etc/ndo.cfg or from /usr/local/nagiosxi/html/config.inc.php
db_host
db_port
db_user
db_pass
Then run this command to start it up.:
Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented/add it if needed:
Make sure all occurrences of this line are commented:
Then start the nagios service:
Then apply configuration and monitor it.
While likely unrelated, edit this file:
Code: Select all
/usr/local/nagiosxi/html/config.inc.phpCode: Select all
$cfg['use_https'] = false;Code: Select all
$cfg['use_https'] = true;Code: Select all
Total Hosts: 7591
Total Services: 46661Generally at 10K total combined host/service checks we recommend that you setup a RAMDisk (which you've done), and at around 20K we recommend you start looking at adding an additional XI server because they can only process so much. Now this may come sooner or later than 20K depending on what type of checks you are running, how much resources they use, your hardware speed, and what you're doing to mitigate the impact.
You should run this check profiler script and see what long running checks you have and determine what some of your long running checks are, they consume resources the whole time they are running so reducing those helps a lot:
https://exchange.nagios.org/directory/P ... me/details
Please read through this doc as well, with the number of checks you are running I would leave the DB local though at this point in time because of the large amount of total checks you have, it requires a lot of throughput to the DB (recommended enabling jumbo_frames):
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
You can only do so much on a single server, you'll need to do what you can to mitigate the impact but you should start looking at adding another XI server soon if you continue to experience load/performance issues after doing the mitigation.
Let me know if you have any questions or if I can clarify anything.
I did have a customer say he had better performance with the old NDO2DB, if you'd like to try that to see if it helps (I would):
Run these commands as root:
Code: Select all
systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db- You can get the info from your /usr/local/nagios/etc/ndo.cfg or from /usr/local/nagiosxi/html/config.inc.php
db_host
db_port
db_user
db_pass
Then run this command to start it up.:
Code: Select all
systemctl start ndo2dbCode: Select all
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfgCode: Select all
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfgCode: Select all
systemctl start nagios-
centricrm
- Posts: 29
- Joined: Tue Jun 05, 2018 3:58 am
Re: Slow Nagios server
Hi,
All servers are upgraded to 5.8.3. At first there is no improvement on the first server, but we will see for some days.
Please find attached a graph of the 5min active checks.
Maybe we can schedule an online session to see the issue for yourself.
wkr
Humphrey
All servers are upgraded to 5.8.3. At first there is no improvement on the first server, but we will see for some days.
Please find attached a graph of the 5min active checks.
Maybe we can schedule an online session to see the issue for yourself.
wkr
Humphrey
You do not have the required permissions to view the files attached to this post.
-
ssax
- Dreams In Code
- Posts: 7682
- Joined: Wed Feb 11, 2015 12:54 pm
Re: Slow Nagios server
Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:
https://support.nagios.com/tickets/
Attach a fresh copy of your profile from the upgraded system in that ticket as well.
Thank you!
https://support.nagios.com/tickets/
Attach a fresh copy of your profile from the upgraded system in that ticket as well.
Thank you!