xi 2011R3.2 Mysql Crash

This support forum board is for support questions relating to Nagios xi, our flagship commercial network monitoring solution.
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: xi 2011R3.2 Mysql Crash

Post by mguthrie »

I forget if the query runs twice, or if the generate_sql_query function processes the LIMIT filter at the very end, but that alone shouldn't cause the issue. My test I did this morning was at over 16k notifications in the table, and it completed without the mod I sent, although it took almost 3 minutes. With the GROUP BY statement removed the page loaded in about 1 second.

What kind of output do you get from running:

Code: Select all

/usr/local/nagiosxi/cron/dbmaint.php
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

Here you go.....

Code: Select all

CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1319400148)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1319400148)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1350331348)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1343160148)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1343160148)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1343160148)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1343160148)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1287864148)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1350935848)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1350935848)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1350935848)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1350935848)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1350935848)
LASTOPT:  1350936001
INTERVAL: 60
NOW:      1350936148
OPTTIME:  1350939601
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < 1350907348::abstime::timestamp without time zone
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < 1350907348::abstime::timestamp without time zone
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
SQL2: DELETE FROM xi_meta WHERE meta_id IN (SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL)
CLEANING nagiosxi TABLE 'auditlog'...
SQL: DELETE FROM xi_auditlog WHERE log_time < 1348344148::abstime::timestamp without time zone
CLEANING nagiosql TABLE 'logbook'...
SQL: DELETE FROM tbl_logbook WHERE time < FROM_UNIXTIME(1350907348)
Repair Complete: Removing Lock File
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

Just to add the reason why its puzzling is because we have got another xi with a much larger notfications history ...and that loads almost instantly on clicking...!
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: xi 2011R3.2 Mysql Crash

Post by mguthrie »

Are there version differences between any of these servers?

Can you give a breakdown of what xi version you're running on each, along with what version of mysql each server has?
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

I did post it right at the beginning however no battles lost :)

Server working super well

1. Centos 5.8
2. xi Version 2011R3.1
3. Mysql Version 5.0.18
4. DB Records in notifications table is about 44k
5. Generated report for a quarter- Took about 2 minutes


Server gone dead!!!

1. Centos 6.3
2. xi Version 2011R3.2
3. Mysql version 5.1.61
4. DB Records in notifications table is about 7k
5. Generated Report for 24 hours about 7 minutes, takes almost the sametime to go from page 1 to page 2

I am sending you a copy of the mysql backup offline..
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

I have PM'd you the link...
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: xi 2011R3.2 Mysql Crash

Post by mguthrie »

Ok, got it.
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

Hi Mike,

I have a quick update on this one for you, today I took a backup of the problematic xi box and restored it on a completely different Hardware install running Centos 5.8 and Mysql 5.0.18 and the problem still persists. So we can now focus on the contents of the DB tables....

Many Thanks
Arnab
mguthrie
Posts: 4380
Joined: Mon Jun 14, 2010 10:21 am

Re: xi 2011R3.2 Mysql Crash

Post by mguthrie »

Is a remote session a possibility in your environment?
User avatar
arnab.roy
Posts: 354
Joined: Sat Apr 30, 2011 10:24 am

Re: xi 2011R3.2 Mysql Crash

Post by arnab.roy »

HI Mike,

Yes not a problem at all, i will setup a remote session. You let me know a time ...

Cheers
Arnab