High load average on xi

This support forum board is for support questions relating to Nagios xi, our flagship commercial network monitoring solution.
cikoski
Posts: 13
Joined: Wed Sep 19, 2012 1:55 am

High load average on xi

Post by cikoski »

Hi,

i have a big problem with my nagios xi server, the load average is too much high: CRITICAL - load average: 40.75, 39.76, 39.11

how can i try to decrease it?

Thanks !
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: High load average on xi

Post by mguthrie »

Usually when it spikes that high there is some sort of database corruption. Can you post the output from running "top"?

http://assets.nagios.com/downloads/nagi ... tabase.pdf
cikoski
Posts: 13
Joined: Wed Sep 19, 2012 1:55 am

Re: High load average on xi

Post by cikoski »

mguthrie wrote:Usually when it spikes that high there is some sort of database corruption. Can you post the output from running "top"?

http://assets.nagios.com/downloads/nagi ... tabase.pdf
I already o the repairing database action, it's work fine and it repair all the tables, it take me 1h for do that.

I have 3000 services and 65 hosts..
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: High load average on xi

Post by scottwilkerson »

How many CPU's does this server have? Did the database repair drop the load?
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
cikoski
Posts: 13
Joined: Wed Sep 19, 2012 1:55 am

Re: High load average on xi

Post by cikoski »

scottwilkerson wrote:How many CPU's does this server have? Did the database repair drop the load?
I have a VM CPU 2 cores and 4GB of RAM, the repair don^t drop the load :D
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: High load average on xi

Post by scottwilkerson »

I think you are going to need to do a bit of investigating as to which processes are using the CPU on your system.

You are running with about 1/2 the resources recommended for that many hosts/services if you are checking them on a 5 minute interval
http://assets.nagios.com/downloads/nagi ... ements.pdf

If you would like you can PM me and I would be willing to do a short remote session to see if we can help uncover what is causing the load.
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
cikoski
Posts: 13
Joined: Wed Sep 19, 2012 1:55 am

Re: High load average on xi

Post by cikoski »

scottwilkerson wrote:I think you are going to need to do a bit of investigating as to which processes are using the CPU on your system.

You are running with about 1/2 the resources recommended for that many hosts/services if you are checking them on a 5 minute interval
http://assets.nagios.com/downloads/nagi ... ements.pdf

If you would like you can PM me and I would be willing to do a short remote session to see if we can help uncover what is causing the load.
Ok, thanks so much for the support, one ask.. How i can watch which processes are using the CPU in my system?

Thanks!
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: High load average on xi

Post by mguthrie »

Run the following command:

Code: Select all

top
Then press 'shift+f', and then 'l' to sort by CPU time. Go ahead and post a screenshot of what you're seeing.

I'm wondering if there's still corruption in the mysql DB even after the repair run. Try the following:

Code: Select all

service mysqld stop
myisamchk -r -f /var/lib/mysql/nagios/*.MYI
service mysqld start



And then lets also make sure postgres is in good shape:

Code: Select all

psql nagiosxi nagiosxi
vacuum;
vacuum analyze;
vaccum full;
\q

psql postgres postgres
vacuum;
vacuum analyze;
vaccum full;
\q
Last edited by mguthrie on Thu Sep 20, 2012 9:28 am, edited 1 time in total.
Reason: More info
cikoski
Posts: 13
Joined: Wed Sep 19, 2012 1:55 am

Re: High load average on xi

Post by cikoski »

mguthrie wrote:Run the following command:

Code: Select all

top
Then press 'shift+f', and then 'l' to sort by CPU time. Go ahead and post a screenshot of what you're seeing.

I'm wondering if there's still corruption in the mysql DB even after the repair run. Try the following:

Code: Select all

service mysqld stop
myisamchk -r -f /var/lib/mysql/nagios/*.MYI
service mysqld start



And then lets also make sure postgres is in good shape:

Code: Select all

psql nagiosxi nagiosxi
vacuum;
vacuum analyze;
vaccum full;
\q

psql postgres postgres
vacuum;
vacuum analyze;
vaccum full;
\q
Thanks so much for the answers, now i take the screenshot..
You do not have the required permissions to view the files attached to this post.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises

Re: High load average on xi

Post by scottwilkerson »

Actually this looks like it is sorted by TIME
Run again and use SHIFT < or > to make the CPU the sorted field
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart