Email alert notification issues with Nagios xi 5.7.4

This support forum board is for support questions relating to Nagios xi, our flagship commercial network monitoring solution.
mejokj
Posts: 353
Joined: Mon Jul 22, 2013 10:31 pm

Email alert notification issues with Nagios xi 5.7.4

Post by mejokj »

Hi Team,

We have recently upgraded Nagiosxi to 5.7.4 and are facing an issue with alert notification. Every alternate day, we won't receive any email alert notification. After doing a "Monitoring Engine" restart it will start again sending an alert notification to the email. This issue is common to all hosts/services

Please note we don't see any entry for alert under incident management--> notification page until a restart of the monitoring engine.

We could notice the Table './nagios/nagios_logentries' is marked as crashed and it has been repaired.

Also ran the database repair command but couldn't find any error.

Do we have any bugs in version 5.7.4?

=====================================================================================================
[root@nagios tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_nagios-lv_root 267G 12G 242G 5% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 485M 31M 430M 7% /boot
[root@nagios tmp]#
[root@nagios tmp]#


[root@nagios tmp]# uptime
10:30:36 up 828 days, 8:43, 1 user, load average: 6.08, 6.18, 6.71
[root@nagios tmp]# free -m
total used free shared buffers cached
Mem: 15913 10786 5127 0 173 789
-/+ buffers/cache: 9822 6091
Swap: 8023 1046 6977
[root@nagios tmp]#

================================================================================================

Thanks
You do not have the required permissions to view the files attached to this post.
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Email alert notification issues with Nagios xi 5.7.4

Post by dchurch »

I'm thinking the emails are getting stuck in a queue. If you go to Admin (top menu) => Email Settings (left menu) and turn on logging, it may generate some useful log data.

Questions:
1. Are you using SMTP or sendmail? (you can view this setting on the Email Settings page)
2. What's the output from the "mailq" command?
3. What happens when you run "postqueue -f"

If you PM me a system profile I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.

If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:

Code: Select all

rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORT
Then send me the resulting /usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
mejokj
Posts: 353
Joined: Mon Jul 22, 2013 10:31 pm

Re: Email alert notification issues with Nagios xi 5.7.4

Post by mejokj »

Hi Team,

Profile file sent to you as PM.

Questions:
1. Are you using SMTP or sendmail? (you can view this setting on the Email Settings page)

>>>> We are using SMTP

2. What's the output from the "mailq" command?

[root@nagios tmp]# mailq
Mail queue is empty
[root@nagios tmp]#


3. What happens when you run "postqueue -f"

[root@nagios tmp]# postqueue -f
[root@nagios tmp]#
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Email alert notification issues with Nagios xi 5.7.4

Post by dchurch »

The root mailbox is full on your server. Run this command:

Code: Select all

>/var/mail/root
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
mejokj
Posts: 353
Joined: Mon Jul 22, 2013 10:31 pm

Re: Email alert notification issues with Nagios xi 5.7.4

Post by mejokj »

Hi dchurch,

We have cleared the root mailbox (>/var/mail/root) .but still the issue happened again.

Could you please check

Thanks,
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Email alert notification issues with Nagios xi 5.7.4

Post by dchurch »

The "nagios_logentries" table may be becoming crashed and sits for a period of time awaiting the automatic database repair script to repair it, at which point the notifications resume.

Try running this command to convert it to the ACID-compliant table type InnoDB. What this will get you is the table will no longer crash, ever but at the expense of some minor performance hit:

Code: Select all

mysql -uroot -pnagiosxi nagios <<< 'alter table nagios_logentries engine = innodb'
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
mejokj
Posts: 353
Joined: Mon Jul 22, 2013 10:31 pm

Re: Email alert notification issues with Nagios xi 5.7.4

Post by mejokj »

Hi dchurch,

Thanks for the update.

Any specific reason for the "nagios_logentries" table become crash very often as it usually won't crash in normal condition.

Thanks
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Email alert notification issues with Nagios xi 5.7.4

Post by ssax »

Please PM a copy of your profile.

Go to Admin > Performance Settings and increase ALL THREE Optimize Intervals to 300.

Additionally, please send the output of these commands:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
This next command may fail, that's okay, not all systems run postgresql, send the output anyways:

Code: Select all

echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi
Include the output of this command:

Code: Select all

sar