xi: adjustments to support large deployments & APP backups

This support forum board is for support questions relating to Nagios xi, our flagship commercial network monitoring solution.
User avatar
inversecow
Posts: 44
Joined: Wed Sep 25, 2019 4:17 pm

xi: adjustments to support large deployments & APP backups

Post by inversecow »

Ahoy folks,

We maintain several Nagios xi instances within our ENVs (some large deployments, some small).

For most of our ENVs, we can rely on either the built-in xi level "Scheduled Backups (Local Backups)" facility.
This works well (for the most part), in that it seems to have a built-in "syslog rotate" type facility (so as to keep the backup store from becoming burdensome).
That said, it would be nice to book for more than one backup (EG: ideal would be twice daily, before start of business & after close of business).

Anyway, while all this works well for our small && medium sized deployments, this does not work for our large deployment.
Specifically, the xi level backup facility strikes upon some sort of built-in "backup timeout" value, and bails.
Most likely this is due to the quantity of data being backed up.

Interestingly, this does not appear to be a problem for manually initiated CLI level backups (via `/usr/local/nagiosxi/scripts/backup_xi.sh`).
It of course just takes a really long time (again due to size of data being backed up).

In reviewing the forums, I see mention of some tuning parameters available (EG: backup_timeout).
I also recall seeing a note that for large deployments, additional tuning was required (not sure if config.inc.php, or php.ini)?

For sake of calibration, `/usr/local/` in our large deployment tends to average around 100GB at present.
This covers ~4.2K ACTIVE checks (host up/down via check_icmp), and ~42,300 PASSIVE checks (via NRDP and check_dummy).
That mount is on its own SAN disk, and we are in the process of working in RAMDisks in our ENVs (in progress).

Thoughts on how we might tune the APP parameters to permit backups against ~100GB+ data mount for xi installations?
Perhaps it would be better to switch to initiating CLI level backups (EG: via cron or scheduled ansible execution), with some sort of built-in clean-up (purge anything older than X days)?

Nagios xi 5.8.x release series (5.8.6 as of tonight)
MariaDB 5.5.64 (off-box for this ENV)

Thank you,

- Rowan
“And who better understands the Unix-nature?” Master Foo asked.
“Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”
Master Foo - The ten thousand Lines
Unix Koans of Master Foo
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: xi: adjustments to support large deployments & APP backu

Post by benjaminsmith »

Hi Rowan,

I hope you are doing well.

There's a couple of angles here. First I would increase the max execution time in the PHP settings. Not sure if that's an issue here or not but we usually recommend increasing the defaults. Please review the following kb article for details and instructions:

Nagios xi - Optimizing The PHP Settings File

Next, try adding the following to the bottom of /usr/local/nagiosxi/html/config.inc.php and then restart the apache and nagios services.

Code: Select all

$cfg['backup_timeout'] = 3600;
If your still having troubles with the scheduled backup not completing, let's check the log and see if there are any errors.

Code: Select all

/usr/local/nagiosxi/var/components/scheduledbackups.log
Regards,
Ben
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!