nagios_servicechecks inserts failed - xi 5.8.2

level2_prod_support · Post by **level2_prod_support** » Mon May 10, 2021 7:32 am

MYSQL insert query failed messages are showing on nagios.log. Could you please guide, how to resolve this issue.

Version: Nagios xi 5.8.3

[1620649194] NDO-3: INSERT INTO nagios_servicechecks (instance_id, start_time, start_time_usec, end_time, end_time_usec, service_object_id, check_type, current_check_attempt, max_check_attempts, state, state_type, timeout, early_timeout, execution_time, latency, return_code, output, long_output, perfdata, command_object_id, command_args, command_line) VALUES (1,FROM_UNIXTIME(1620649194),921580,FROM_UNIXTIME(1620649194),927575,1026,0,1,1,0,1,60,0,0.005995,0.000928,0,'D:\\ - total: 20.00 Gb - used: 2.19 Gb (11%) - free 17.81 Gb (89%)','','\'D:\\ Used Space\'=2.19Gb;16.00;19.00;0.00;20.00',0,'','') ON DUPLICATE KEY UPDATE instance_id = VALUES(instance_id), start_time = VALUES(start_time), start_time_usec = VALUES(start_time_usec), end_time = VALUES(end_time), end_time_usec = VALUES(end_time_usec), service_object_id = VALUES(service_object_id), check_type = VALUES(check_type), current_check_attempt = VALUES(current_check_attempt), max_check_attempts = VALUES(max_check_attempts), state = VALUES(state), state_type = VALUES(state_type), timeout = VALUES(timeout), early_timeout = VALUES(early_timeout), execution_time = VALUES(execution_time), latency = VALUES(latency), return_code = VALUES(return_code), output = VALUES(output), long_output = VALUES(long_output), perfdata = VALUES(perfdata), command_object_id = VALUES(command_object_id), command_args = VALUES(command_args), command_line = VALUES(command_line)
[1620649195] NDO-3: The following query failed while MySQL appears to be connected:

gsmith · Post by **gsmith** » Mon May 10, 2021 3:23 pm

Hi there,

Could you please get me a System Profile?

To send us your system profile.
1. Login to the Nagios xi GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Additionally on a CLI of the server please run:
mysqlcheck -u root -p nagios
( default password is nagiosxi)

Let me know if anything comes back without an "OK" result.

Thanks

level2_prod_support · Post by **level2_prod_support** » Mon May 10, 2021 8:21 pm

-bash-4.2$ mysqlcheck -u root -p nagios
Enter password:
nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_conninfo OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands OK
nagios.nagios_flappinghistory OK
nagios.nagios_host_contactgroups OK
nagios.nagios_host_contacts OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members OK
nagios.nagios_hostgroups OK
nagios.nagios_hosts OK
nagios.nagios_hoststatus OK
nagios.nagios_instances OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects OK
nagios.nagios_processevents OK
nagios.nagios_programstatus OK
nagios.nagios_runtimevariables OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services OK
nagios.nagios_servicestatus OK
nagios.nagios_statehistory OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges OK
nagios.nagios_timeperiods OK

gsmith · Post by **gsmith** » Tue May 11, 2021 1:05 pm

Hi

Your database was having problems because your disk is full:

Code: Select all

210313  8:36:35 [Note] Retrying repair of: './nagios/nagios_statehistory' including modifying data file
210315 12:40:05 [Warning] IP address '172.30.12.134' could not be resolved: Name or service not known
210319  1:10:15 [Warning] mysqld: Disk is full writing './nagios/nagios_logentries.TMD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210319  1:10:15 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210326  1:10:31 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210326  1:10:31 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210327  1:09:50 [Warning] mysqld: Disk is full writing './nagios/nagios_eventhandlers.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210327  1:09:50 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210329  1:08:35 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210329  1:08:35 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210330  1:08:53 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210330  1:08:53 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210330  9:46:04 [Warning] IP address '172.30.10.2' could not be resolved: Name or service not known
210401  1:08:23 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_contactnotificationmethods.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210401  1:08:23 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210402  1:07:58 [Warning] mysqld: Disk is full writing './nagios/nagios_statehistory.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210402  1:07:58 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210403  1:08:43 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210403  1:08:43 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210404  1:08:52 [Warning] mysqld: Disk is full writing './nagios/nagios_eventhandlers.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
210404  1:08:52 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs
210405  1:08:26 [Warning] mysqld: Disk is full writing '/var/lib/mysql/nagios/nagios_logentries.MYI' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing dis

Take care of that and let me know if the insert problem is solved.

Thanks

level2_prod_support · Wed May 12, 2021 11:22 am

/var has enough space. I don't see disk full on the box. Could you please guide which one to clear.

-bash-4.2$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-varvol 16G 9.6G 4.9G 67% /var
-bash-4.2$

-bash-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 98M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/rootvg-rootvol 16G 4.2G 11G 29% /
/dev/sda1 969M 113M 791M 13% /boot
/dev/mapper/rootvg-varvol 16G 9.6G 4.9G 67% /var
/dev/mapper/storevg-store 70G 5.7G 65G 9% /store
/dev/mapper/userlocalvg-usrlocal 50G 19G 32G 38% /usr/local
/dev/mapper/appsvg-apps 25G 40M 25G 1% /apps
tmpfs 6.3G 0 6.3G 0% /run/user/45128

+--------------------+--------------+
| table_schema | MB |
+--------------------+--------------+
| information_schema | 0.15625000 |
| mysql | 2.52234554 |
| nagios | 395.44248578 |
| nagiosql | 3.50000000 |
| nagiosxi | 53.00000000 |
| performance_schema | 0.00000000 |
| sys | 0.01562500 |
+--------------------+--------------+
7 rows in set (0.06 sec)

gsmith · Post by **gsmith** » Wed May 12, 2021 11:32 am

Hi

The logs were showing /var as being full, but somehow that was mitigated. Please take another
System Profile and send it to me. Thanks

To send us your system profile.
1. Login to the Nagios xi GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

level2_prod_support · Post by **level2_prod_support** » Wed May 12, 2021 1:40 pm

Sent profile file as PM. please review

gsmith · Post by **gsmith** » Thu May 13, 2021 10:19 am

Hi

The profile was not attached. Please resend.

Thanks

level2_prod_support · Thu May 13, 2021 12:07 pm

Sent profile file again as PM. please review

gsmith · Post by **gsmith** » Thu May 13, 2021 4:22 pm

hi,

Looks good. Please run the insert command manually and let me know the output.

So it would be something like:

Code: Select all

mysql -u root -p nagios

<enter password>

 INSERT INTO nagios_servicechecks (instance_id, start_time, start_time_usec, end_time, end_time_usec, service_object_id, check_type, current_check_attempt, max_check_attempts, state, state_type, timeout, early_timeout, execution_time, latency, return_code, output, long_output, perfdata, command_object_id, command_args, command_line) VALUES (1,FROM_UNIXTIME(1620844440),0,FROM_UNIXTIME(0),0,2028,1,1,1,0,1,60,1,0.000000,0.000000,0,'Process UP=6','','',0,'','') ON DUPLICATE KEY UPDATE instance_id = VALUES(instance_id), start_time = VALUES(start_time), start_time_usec = VALUES(start_time_usec), end_time = VALUES(end_time), end_time_usec = VALUES(end_time_usec), service_object_id = VALUES(service_object_id), check_type = VALUES(check_type), current_check_attempt = VALUES(current_check_attempt), max_check_attempts = VALUES(max_check_attempts), state = VALUES(state), state_type = VALUES(state_type), timeout = VALUES(timeout), early_timeout = VALUES(early_timeout), execution_time = VALUES(execution_time), latency = VALUES(latency), return_code = VALUES(return_code), output = VALUES(output), long_output = VALUES(long_output), perfdata = VALUES(perfdata), command_object_id = VALUES(command_object_id), command_args = VALUES(command_args), command_line = VALUES(command_line)

Let me know if it works or not.
If you get an error running the above insert statement then save the error message for me and then I want you to
downgrade the NDO-3 broker to ndoutils by following these steps:

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nag ... .14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./installsystemctl enable ndo2db
systemctl start ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is not commented out or if it is not there add it:
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Search for the line below and comment it out by adding a # at the beginning of the line:
#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service: systemctl start nagios

Now try running the sql insert statement again.

Let me know all the results please.

Thanks

Nagios Support Forum

nagios_servicechecks inserts failed - xi 5.8.2

nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2

Re: nagios_servicechecks inserts failed - xi 5.8.2