I ran into a problem yesterday when I went to take a backup before updating our Nagios xi install with the backup failing with a permissions error. This was particularly surprising because it is configured to take a scheduled backup every day, and to email on failures, and we had not gotten any emails for failures (until the one we got when the manual backup I ran failed). I tried running the backup script from command line and got the same error.
This was the backup output:
\nStarting new backup....\n
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
Backing up Nagios xi...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up the SNMP directories
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up nagios user home dir...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces
Backing up cronjobs for Apache...
Backing up logrotate config files...
Backing up Apache config files...
Compressing backup...
===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/1617216057.tar.gz
After a bit of digging, I granted the process permission to each of the three users the backup script appeared to be using to connect to the MySQL database, and then retried the backup script. It no longer encountered the mysqldump permissions error, however it still emailed that the backup failed, even though the closest thing to an error in the output was the warning messages about using passwords with the command line interface. What's even stranger, is that when the scheduled backup ran later that day, it did not send a failure email, even though the output from it was identical.
I'm a bit lost on why it does not seem to be sending the email on failure for the scheduled backup, why it is still sending the email for failure when run manually when it does not seem to be failing anymore, and why it started to or was getting the permissions error to begin with. Also on a side note, is the output from the backup script archived anywhere when the script runs or is it really just overwriting the same file every time? It would be nice to be able to go through an actual log to see when it started getting this error or if it was always getting it.
Any insight or guidance is appreciated.
Nagios xi Backup
-
vtrac
- Posts: 903
- Joined: Tue Oct 27, 2020 1:35 pm
Re: Nagios xi Backup
Hi,
I have just tried the "/usr/local/nagiosxi/scripts/backup_xi.sh" backup script as "root" and it completed just fine.
What was the "user" you used (logged in) to run this script?
Also, can you please show me the complete command you used? ..... did you use "sudo"?
I'm not sure what OS you are on, but could you please check the database's log and see if there is any crashed table that need repair.
Log location:
If your database crashed, please run this command:
Also, check the below logs after or during the running of the backup script with the "tail -f" command:
If you still having issue running the backup script, please upload the "profile.zip".
Regards,
Vinh
I have just tried the "/usr/local/nagiosxi/scripts/backup_xi.sh" backup script as "root" and it completed just fine.
What was the "user" you used (logged in) to run this script?
Also, can you please show me the complete command you used? ..... did you use "sudo"?
I'm not sure what OS you are on, but could you please check the database's log and see if there is any crashed table that need repair.
Log location:
Code: Select all
/var/log/mariadb/mariadb.log
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.sh
Code: Select all
/usr/local/nagios/var/nagios.log
and
/usr/local/nagiosxi/var/cmdsubsys.log
If you still having issue running the backup script, please upload the "profile.zip".
Regards,
Vinh
-
ot_sysadmin
- Posts: 6
- Joined: Tue Jan 19, 2021 1:43 pm
Re: Nagios xi Backup
The script was run as root (run using sudo).vtrac wrote:What was the "user" you used (logged in) to run this script?
Also, can you please show me the complete command you used? ..... did you use "sudo"?
Full command:
sudo /usr/local/nagiosxi/scripts/backup_xi.sh
It's worth noting that I first got the error running the script from the web UI (admin --> system backups --> local backup archives --> create backup). After that I tried running the script via command line to see if it was any different, but got the same error there as well.
The host is running CentOS 8. I do not see any errors being written to the database logs (/var/log/mysql/mysqld.log). I restarted both the Nagios processes and mysqld anyway, and now I am only getting the failure email when running the backup via the web UI.
I ran the backup via the web UI while tailing the files you mentioned. I did not see any errors in the nagios.log. I have attached the output from the cmdsubsys.log that I saw for the backup. It stated that the process ended with a return code of 0...
I have also attached the profile.zip.
You do not have the required permissions to view the files attached to this post.
-
vtrac
- Posts: 903
- Joined: Tue Oct 27, 2020 1:35 pm
Re: Nagios xi Backup
Hi,
From what I see, your backup completed just fine.
The below were just warnings, not errors.
Also, is your database local?
I do not see the database log in the "profile.zip".
What is the output of the below command on your xi machine?
Regards,
Vinh
From what I see, your backup completed just fine.
The below were just warnings, not errors.
Code: Select all
Backing up MySQL databases...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
..mysqldump: [Warning] Using a password on the command line interface can be insecure.
...mysqldump: [Warning] Using a password on the command line interface can be insecure.
Backing up cronjobs for Apache...
I do not see the database log in the "profile.zip".
What is the output of the below command on your xi machine?
Code: Select all
ps -ef | grep mysql
Regards,
Vinh
-
ot_sysadmin
- Posts: 6
- Joined: Tue Jan 19, 2021 1:43 pm
Re: Nagios xi Backup
Hi Vinh,
I agree that based on the output it seems to be completing fine and doesn't seem to be getting any errors anymore, so i'm trying to understand why it is still sending the failure email when run via the web UI.
Yes, the database is local.
$ ps -ef | grep mysql
mysql 3278033 1 3 Apr06 ? 01:00:11 /usr/libexec/mysqld --basedir=/usr
$
I agree that based on the output it seems to be completing fine and doesn't seem to be getting any errors anymore, so i'm trying to understand why it is still sending the failure email when run via the web UI.
Yes, the database is local.
$ ps -ef | grep mysql
mysql 3278033 1 3 Apr06 ? 01:00:11 /usr/libexec/mysqld --basedir=/usr
$
-
vtrac
- Posts: 903
- Joined: Tue Oct 27, 2020 1:35 pm
Re: Nagios xi Backup
Hi,
Did you make changes to mysql config files?
Please upload all mysql config files so I can take a look.
Why did those user's permission changed? .... please investigate why someone changed these user permissions.
From what I know, it was working before.
When I ran the "ps -ef | grep mysql", I got the below outputs.
Checked on couple of my machines (VM), all looked the same:
Also, based on the cmdsubsys.log, it completed fine with a status of "0":
Can you please post the email with the error? ... please post the latest one.
Regards,
Vinh
Did you make changes to mysql config files?
Please upload all mysql config files so I can take a look.
Based on the statement above, what are the three users you have granted access to.After a bit of digging, I granted the process permission to each of the three users the backup script appeared to be using to connect to the MySQL database, and then retried the backup script
Why did those user's permission changed? .... please investigate why someone changed these user permissions.
From what I know, it was working before.
When I ran the "ps -ef | grep mysql", I got the below outputs.
Checked on couple of my machines (VM), all looked the same:
Code: Select all
[root@vt-nagiosxi-62 send_nrdp_perl]# ps -ef | grep mysql
mysql 1072 1 0 09:02 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql 1416 1072 0 09:02 ? 00:02:16 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
Code: Select all
===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/1617720077.tar.gz
OUTPUT=Backup stored in /store/backups/nagiosxi/1617720077.tar.gz
RETURNCODE=0
Regards,
Vinh
-
ot_sysadmin
- Posts: 6
- Joined: Tue Jan 19, 2021 1:43 pm
Re: Nagios xi Backup
Hi,
I did not make any changes to mysql config files.
The 3 users that I granted "process" permissions to were ndoutils, nagiosql, and nagiosxi.
This is the text in the failure email. I did try setting a backup_timeout variable as it mentions, but it did not make any difference.
The scheduled local backup at 2021-04-06 10:41:22 failed.
Last output from the system:
View your full scheduled backup logs at /usr/local/nagiosxi/var/components/scheduledbackups.log to find out more details about the error. Your backup may have timed out, you can set a backup_timeout variable in config.inc.php to increase it.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
I did not make any changes to mysql config files.
The 3 users that I granted "process" permissions to were ndoutils, nagiosql, and nagiosxi.
This is the text in the failure email. I did try setting a backup_timeout variable as it mentions, but it did not make any difference.
The scheduled local backup at 2021-04-06 10:41:22 failed.
Last output from the system:
View your full scheduled backup logs at /usr/local/nagiosxi/var/components/scheduledbackups.log to find out more details about the error. Your backup may have timed out, you can set a backup_timeout variable in config.inc.php to increase it.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
-
vtrac
- Posts: 903
- Joined: Tue Oct 27, 2020 1:35 pm
Re: Nagios xi Backup
Hi,
Could you please upload these logs (below):
Let see if I understand the issue here (please correct me, if I'm wrong):
- You are able to run the "backup_xi.sh" script manually.
- Your scheduled backup failed.
Also, could you please upload the "/etc/sudoer" file.
I'm guessing that these user (apache or nagios) night not have correct sudo privileges to run the backup script ... just a guess ...
Regards,
Vinh
Could you please upload these logs (below):
Code: Select all
/usr/local/nagiosxi/var/components/scheduledbackups.log
/usr/local/nagiosxi/var/cmdsubsys.log
/var/log/mariadb/mariadb.log
or
/var/log/mysql/mysql.log
- You are able to run the "backup_xi.sh" script manually.
- Your scheduled backup failed.
Also, could you please upload the "/etc/sudoer" file.
I'm guessing that these user (apache or nagios) night not have correct sudo privileges to run the backup script ... just a guess ...
Regards,
Vinh
-
ot_sysadmin
- Posts: 6
- Joined: Tue Jan 19, 2021 1:43 pm
Re: Nagios xi Backup
I tested out restoring from one of the "failed" backups and it restored fine, so I went ahead and tried updating Nagios xi to the latest release to see if that would fix whatever the problem is, but no luck. I have decided to open a formal support ticket to try to get this issue resolved, thanks for all the help!
-
vtrac
- Posts: 903
- Joined: Tue Oct 27, 2020 1:35 pm
Re: Nagios xi Backup
Update: support ticket opened
https://support.nagios.com/tickets/scp/ ... p?id=13829
locking thread!!
https://support.nagios.com/tickets/scp/ ... p?id=13829
locking thread!!