Nagios xi - Schedulded Backups No Longer Working

Problem Description

When looking at the location that the scheduled backups are stored in, there are no new xxxxxxx.tar.gz files being created in this location.

Additionally,  in the /store/backups/nagiosxi/ location there are lots of folders that are not being removed as per the Backup Limit setting.

Explanation

The root cause of the issue is that the backups are failing. When this happens the following occurs:

• The backup does not complete, hence a xxxxxxx.tar.gz file is not created

• The temporary /store/backups/nagiosxi/xxxxxxx folder is not deleted

  • These folders will not be cleaned up as part of the backup limit setting, they will need to be manually deleted

The most common cause of the backup failing is due to a database corruption.

Resolving The Problem

To identify why the backups are failing, you will need to execute a manual backup in an SSH session on your Nagios xi server. This will generate verbose output that will help determine why the backup is failing.

The procedure for performing a manual backup is explained in this document:

Documentation - Backing Up And Restoring Your Nagios xi System

As mentioned before, the most common cause of failing backups is a corrupt database. Here is an example of a failed backup due to a corrupt database:

Running configuration check...
Stopping nagios: done.
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
Backing up Nagios xi...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: Got error: 130: Incorrect file format 'nagios_flappinghistory' when using LOCK TABLES
Error backing up MySQL database 'nagios' - check the password in this script!

The error "check the password in this script" is a generic message and is not the cause, the line before it explains the issue in this example.

Please follow this article to repair the database:

Documentation - Crashed Database Tables

Once the database has been repaired, run a manual backup to confirm that the issue has been resolved.

To remove the directories of failed backup files, execute the following command (this will not touch any .tar.gz files):

find /store/backups/nagiosxi/ -mindepth 1 -maxdepth 1 -type d -exec rm -rf '{}' \;

Watching Scheduled Backups

If you need to see what is actually occurring when the scheduled backup is run, you can watch the output live by executing the following command:

tail /usr/local/nagiosxi/var/cmdsubsys.log -f

Press CTRL + C when you have finished.

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/