Nagios xi performance

paul.jobb · Post by **paul.jobb** » Thu Jan 19, 2012 4:43 pm

Hi

My configuration is as follows:

1 dnx master and 2 dnx slaves, all are VM's with one virtual CPU and 2 GB ram allocated. I have 1444 hosts loaded with 8667 service test,1444 of those are ping tests and the remainder are WMI-plus queries(cpu, memory, drive space).

The performance of the dnx master system is very sluggish, the load 1/5/15 min load is anywhere between 40 and 60. From what I can tell from the two slave systems aren't that busy. I have reviewd the maximizing xi performance document and adjusted the reaper variables and AJAX setting per the recommendations to all three systems.

Any suggestions on improving performance?

Would :
-adding an additional two xi slaves
-adding an additional CPU to the master
-more memory
-move the master to a physical server
be of any benefit in improving performance?

Or is there some way I can further tune the Nagios/DNX configuration?

scottwilkerson · Post by **scottwilkerson** » Thu Jan 19, 2012 5:38 pm

paul.jobb wrote:Would :
-adding an additional two xi slaves
-adding an additional CPU to the master
-more memory
-move the master to a physical server
be of any benefit in improving performance?

All of these listed would help, but you haven't specified any other high performance problems other than load. This would lead me to believe that the problem could be I/O disk access.

So Moving to a physical server especially one with raid drives. Also though, you could pick up some performance with a Ram Disk setup

mguthrie · Post by **mguthrie** » Thu Jan 19, 2012 5:43 pm

I was just actually doing some performance tests today. One thing I would mention is that if you don't have mysql offloaded onto a separate machine, that makes a HUGE difference. The other thing I'm discovering for scaling xi to a larger install is that disk I\O is the number one bottleneck for hardware. Here are some other docs we've created in addition to the Maximizing xi performance:

http://assets.nagios.com/downloads/nagi ... p#boosting

WMI checks are fairly CPU intensive, so they will put a big hit on whatever machine is running them.

I also just experimented with moving the /usr/local/nagios/var/spool directories to a RAM disk as well as objects.cache and status.dat files. Here are some other resources.
http://exchange.nagios.org/directory/Mu ... ny/details

http://exchange.nagios.org/directory/Mu ... ks/details

paul.jobb · Post by **paul.jobb** » Fri Jan 20, 2012 11:38 am

Thanks for the Reply

Attached on the bottom of this email is some additional statistics as to current load of the server. Navigating the web interface is slow expecially trying to look at service and host views, those pages will often time out. I will definitely look into the RAM disk configuration, would like to avoid using a physical machine if I can.
Couple questions

In regards to the wmi queries being CPU intesive, would that suggest then that these tests aren't being pushed to my DNX slaves then? When running the top command on the slave machines I do see php processes running for the Nagios account, is there anyway to confirm those are wmi queries and not just pings?

Is there any documentaion as to how to setup and configure Nagios to use an offloaded mysql database? would that just be in the standard documentation?

Code: Select all

Nagios xi

Server Stats
Server Statistics
1-min	57.06	
5-min	54.33	
15-min	51.77	
 
CPU Stats
User	91.20%	
Nice	0.00%	
System	8.80%	
I/O Wait	0.00%	
Steal	0.00%	
Idle	0.00%	
 
Memory
Total	2023 MB
Used	1550 MB	
Free	472 MB	
Shared	0 MB	
Buffers	90 MB	
Cached	422 MB	
 
Swap
Total	255 MB
Used	1 MB	
Free	254 MB	
 


Monitoring Engine Performance
Metric Value  
Host Check Latency 
Min 0.00 sec   
Max 1145.99 sec   
Avg 1074.65 sec   
Host Check Execution Time 
Min 0.01 sec   
Max 10.76 sec   
Avg 0.44 sec   
Service Check Latency 
Min 1050.52 sec   
Max 1149.46 sec   
Avg 1101.97 sec   
Service Check Execution Time 
Min 0.03 sec   
Max 28.40 sec   
Avg 9.53 sec   


Monitoring Engine Check Statistics
Metric Value  
Active Host Checks 
1-min 61   
5-min 378   
15-min 654   
Passive Host Checks 
1-min 0   
5-min 0   
15-min 0   
Active Service Checks 
1-min 177   
5-min 1748   
15-min 3080   
Passive Service Checks 
1-min 0   
5-min 0   
15-min 0

mguthrie · Post by **mguthrie** » Fri Jan 20, 2012 1:13 pm

I think there's something else wrong on that system. The CPU load is way too high for the system to be healthy. I'm betting you have corruption in either mysql or postgresql.

Mysql
http://assets.nagios.com/downloads/nagi ... tabase.pdf

Postgresl:

Code: Select all

psql nagiosxi nagiosxi
vacuum;
vacuum analyze;
vacuum full;

Have you turned on any additional logging with PNP or the NPCD daemon?

paul.jobb · Post by **paul.jobb** » Fri Jan 20, 2012 1:37 pm

Thanks I will execute those scripts and see what happens. The server was installed within the past few weeks, and outside of using the bulk load facility to load those 1444 hosts, and adding them to a windows-servers group which I have those wmi service tests configured against, we haven't really done anything fancy. We just used the VM appliance available for download and copied it to our VM environment.

We would very much like to seperate the Database from the xi server if we could though, so if there is any documentation that might assist me with that.

scottwilkerson · Post by **scottwilkerson** » Fri Jan 20, 2012 1:47 pm

Here you go
http://assets.nagios.com/downloads/nagi ... Server.pdf

paul.jobb · Post by **paul.jobb** » Fri Jan 20, 2012 5:36 pm

I seem to have locked myself out core config manager after the database move, all other access appears to be fine, I double checked permissions grant for nagiosql on the new mysql server(83.247). This was the setting I made through the interface(page 4 of the documentation) is there method for adjusting via a configuration file?

Error while connecting to database:
[xxx.xxx.83.247] Connection to the database server has failed by reason:
Access denied for user 'nagiosql'@'xxx.xxx.83.246' (using password: YES)

scottwilkerson · Post by **scottwilkerson** » Fri Jan 20, 2012 9:49 pm

Did you do this step from page 2 to verify you would be able to connect from your Nagios xi server?

To make sure we can actually access these databases remotely, use the following from the Nagios server
mysql -u nagios -p'nagios' -h <IP_ADDRESS_OF_MYSQL_SERVER>
mysql -u nagiosql -p'nagiosql' -h <IP_ADDRESS_OF_MYSQL_SERVER>

Also, all the configuration file information is on page 3. Worst case scenerio you should be able to replace these with the .bak file you copied while stepping through the instructions.

mguthrie · Post by **mguthrie** » Mon Jan 23, 2012 10:37 am

You should be to reset the credentials for the CCM through the Admin->Reset Security Credentials page. However, if you're getting just an error message when trying to access the CCM, you can hardcode your DB credentials in the /var/www/html/nagiosql/config/settings.php file.

Nagios Support Forum

Nagios xi performance

Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance

Re: Nagios xi performance