Hanging xi interface after database server swap

YannPingot · Post by **YannPingot** » Tue Sep 22, 2020 12:38 pm

Hello,

Considering our old version of MySQL running on a RHEL 6, we want to upgrade our offloaded database as a try to improve our overall performances as we have recurrent issues with ipcs queues filling up.

A MariaDB 10.5 (latest version) has been installed on a RHEL 7.7, all 3 databases dumped and imported on the new one.

After changing variables in config files and everything restarted Nagios core seems to be working pretty well as well as NDO2DB, but the xi interface is completely unusable, showing partial interface and multiple databases errors if it respond after several seconds/minutes :

Select of database nagiosxi failed. Lost connection to MySQL server during query.
Select of database nagios failed. Lost connection to MySQL server during query.
Select of database nagiosql failed. Lost connection to MySQL server during query.

Also Mysql tuning parameters have been copied to the new database so it should at least run like the old one.
We're using version 5.7.3 with rollbacked NDO to ndo2db (as NDO3 wasn't working, hoping that this new Mariadb version would improve this).

Thanks for your help,
Yann

Post by **jdunitz** » Wed Sep 23, 2020 2:38 pm

This may help:

https://support.nagios.com/kb/article/n ... s-780.html

Let us know what happens!
--Jeffrey

YannPingot · Post by **YannPingot** » Wed Sep 30, 2020 3:07 am

Hello Jeffrey,

Unfortunately it didn't change anything so i had to rollback again, the UI is still completely unusable, but the core + ndo2db seems to be working fine as it was sending alarms and the ipcs queue unstacking.
The way it works appears to be some kind of timeout, but we can access databases perfectly fine with a mysql client and there is no firewall involved here.
I've tried activating PHP logs but nothing more here (btw we're using PHP 5.3.3).

Any thoughts ?

Thanks,
Yann

ssax · Post by **ssax** » Wed Sep 30, 2020 5:32 pm

Yeah, for MariaDB 10 you'll need to set sql_mode="" from the guide jdunitz posted. Also, try setting max_allowed_packet=512M in your /etc/my.cnf as well.

It shows it's failing for all databases (nagios, nagiosql, nagiosxi) so it's likely the /usr/local/nagiosxi/html/config.inc.php that wasn't proper or there's something else going on.

Actually, did you update this file:

Code: Select all

/usr/local/nagios/etc/ndo2db.cfg

Or this file?

Code: Select all

/usr/local/nagios/etc/ndo.cfg

xi 5.7.2 should use the new NDO by default which would be this one:

Code: Select all

/usr/local/nagios/etc/ndo.cfg

Do you have a copy of your profile from the upgraded state? If not, send a copy of your current one from Admin > System Profile by clicking the Download Profile button.

YannPingot · Post by **YannPingot** » Thu Oct 01, 2020 10:00 am

I've updated ndo2db.cfg as we've rollbacked to NDO2 like is said in my first post.

I've indeed set the sql_mode in the cnf as modifying the parameter through the CLI is not persistent.

max_allowed_packet has been set to 256M as our current DB instance, are you sure increasing it to the double would help ?

I can't retrieve a profile since the UI is totally unresponsive once on the new DB...

Regarding the config.inc.php i've literally just modified the backend address since i've recreated users identically.

ssax · Post by **ssax** » Fri Oct 02, 2020 10:35 am

I would increase it to 512M and see if it helps:

https://dev.mysql.com/doc/refman/5.7/en ... ction.html

Are you seeing any messages on /var/log/messages or /var/log/mariadb/mariadb.log on the offloaded DB server?

If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT

Then send me the resulting /usr/local/nagiosxi/var/components/profile.zip file.

If the profile script fails, please include the ENTIRE output.

YannPingot · Post by **YannPingot** » Wed Oct 07, 2020 11:43 am

Hello Ssax,

I've sent you a PM with the profile attached.

As expected unfortunately increasing the packet size didn't change anything.

Is there a possibility that we could have somewhere else on the web part a cache which could cause problems like that ?

Thanks,
Best regards,
Yann Pingot

ssax · Post by **ssax** » Thu Oct 08, 2020 11:51 am

You currently are running the old NDO2DB but your /usr/local/nagios/etc/ndo2db.cfg file does not match your /usr/local/nagiosxi/html/config.inc.php for the IP address. Which IP is proper? You'll need to fix one or the other and make sure they all match, then restart the services:
-- I added some cleanup commands as well just in case

Code: Select all

systemctl stop httpd
systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop mod-gearman-worker
systemctl stop gearmand
pkill -9 -u nagios
pkill -9 -u apache
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -f /usr/local/nagiosxi/var/dbmaint.lock
rm -f /usr/local/nagiosxi/var/event_handler.lock
rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagios/var/ndo2db.pid
rm -f /usr/local/nagios/var/ndo2db.sock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /us/local/nagiosxi/var/subsys/ndo2db
rm -f /var/run/nagios/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/httpd/httpd.pid
rm -f /usr/local/nagiosxi/var/subsys/npcd.pid
systemctl start gearmand
systemctl start mod-gearman-worker
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl restart httpd
systemctl restart snmptt

YannPingot · Post by **YannPingot** » Mon Oct 12, 2020 3:37 am

My mistake when i created the profile zip, that's why you have a different IP indeed, but of course on my earlier tries it was matching and the behavior was the same (in that case the web part is correctly pointing to the new database).

I'll try again with all cleanup commands you mentioned (well, with their RHEL6 counterparts

), maybe this is linked to sockets or lock files

ssax · Post by **ssax** » Mon Oct 12, 2020 3:26 pm

Let us know if that resolves it.

If it doesn't, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

Thank you!

Nagios Support Forum

Hanging xi interface after database server swap

Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap

Re: Hanging xi interface after database server swap