Nagios ramdisk full and no performance graphs

hbouma · Post by **hbouma** » Mon Nov 22, 2021 11:49 am

Support file sent via PM.

pbroste wrote: Looking back through we see that you stated that the following directory is not created when the 'ramdisk.service' is started and wanted to touch on this further, by taking a look at the journal output on restarted service.

First restart the 'ramdisk.service' by:
Code: Select all
systemctl restart ramdisk.service
Then take a look at the 'systemctl' status to see if the '/var/nagiosramdisk/spool/xidpe' folder is create or missed?
Code: Select all
systemctl -l status ramdisk --no-pager

I do not see it making a /var/nagiosramdisk/spool/xidpe folder.

Code: Select all

systemctl -l status ramdisk --no-pager
● ramdisk.service - Ramdisk
   Loaded: loaded (/usr/lib/systemd/system/ramdisk.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2021-11-22 11:44:45 EST; 18s ago
  Process: 14008 ExecStart=/usr/bin/chown -R nagios:nagios /var/nagiosramdisk (code=exited, status=0/SUCCESS)
  Process: 14004 ExecStartPre=/usr/bin/mkdir -p -m 775 /var/nagiosramdisk/tmp/var/nagiosramdisk/spool /var/nagiosramdisk/spool/checkresults/var/nagiosramdisk/spool/xidpe /var/nagiosramdisk/spool/perfdata (code=exited, status=0/SUCCESS)
  Process: 14001 ExecStartPre=/usr/bin/mount -t tmpfs -o size=500m tmpfs /var/nagiosramdisk (code=exited, status=0/SUCCESS)
  Process: 13998 ExecStartPre=/usr/bin/mkdir -p -m 775 /var/nagiosramdisk (code=exited, status=0/SUCCESS)
 Main PID: 14008 (code=exited, status=0/SUCCESS)

Nov 22 11:44:45 SERVER systemd[1]: Starting Ramdisk...
Nov 22 11:44:45 SERVER systemd[1]: Started Ramdisk.

pbroste wrote: Also, let us know how 'rrdcached.service' looks as well.
Code: Select all
systemctl -l status rrdcached --no-pager
Thanks,
Perry

Here is the output:

Code: Select all

systemctl -l status rrdcached --no-pager
● rrdcached.service - LSB: start and stop rrdtool caching daemon
   Loaded: loaded (/etc/rc.d/init.d/rrdcached; bad; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-16 07:56:23 EST; 6 days ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/rrdcached.service
           └─21288 /usr/bin/rrdcached -p /var/rrdtool/rrdcached/rrdcached.pid -s nagios -m 0660 -l unix:/var/rrdtool/rrdcached/rrdcached.sock -F -w 900 -z 90 -j /var/rrdtool/rrdcached -b /var/rrdtool/rrdcached

Nov 22 08:56:23 SERVER rrdcached[21288]: started new journal /var/rrdtool/rrdcached/rrd.journal.1637589383.218497
Nov 22 08:56:23 SERVER rrdcached[21288]: removing old journal /var/rrdtool/rrdcached/rrd.journal.1637582183.218593
Nov 22 09:56:23 SERVER rrdcached[21288]: flushing old values
Nov 22 09:56:23 SERVER rrdcached[21288]: rotating journals
Nov 22 09:56:23 SERVER rrdcached[21288]: started new journal /var/rrdtool/rrdcached/rrd.journal.1637592983.218610
Nov 22 09:56:23 SERVER rrdcached[21288]: removing old journal /var/rrdtool/rrdcached/rrd.journal.1637585783.218568
Nov 22 10:56:23 SERVER rrdcached[21288]: flushing old values
Nov 22 10:56:23 SERVER  rrdcached[21288]: rotating journals
Nov 22 10:56:23 SERVER  rrdcached[21288]: started new journal /var/rrdtool/rrdcached/rrd.journal.1637596583.218546
Nov 22 10:56:23 SERVER rrdcached[21288]: removing old journal /var/rrdtool/rrdcached/rrd.journal.1637589383.218497

Post by **pbroste** » Tue Nov 23, 2021 11:16 am

Hello @hbouma

Thanks for sending over the info, from what we can see from the results it appears from the 'ramdisk.service' startup that all directories are created ('/var/nagiosramdisk/spool/xidpe'). Please send the list on the directories after restarting ramdisk.service:

Code: Select all

ls -lahrtR /var/nagiosramdisk/

Thanks,
Perry

hbouma · Post by **hbouma** » Tue Nov 23, 2021 1:00 pm

FYI, the Nagios RAMDISK has started filling up again.

Code: Select all

12:55 PM SERVER root [~]
$ systemctl restart ramdisk.service

ls -lahrtR /var/nagiosramdisk/
/var/nagiosramdisk/:
total 4.0K
drwxr-xr-x. 24 root   root   4.0K Mar 29  2019 ..
drwxr-xr-x   3 nagios nagios   60 Nov 23 12:55 tmp
drwxr-xr-x   4 nagios nagios   80 Nov 23 12:55 spool
drwxrwxrwt   4 nagios nagios   80 Nov 23 12:55 .

/var/nagiosramdisk/tmp:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 var
drwxrwxrwt 4 nagios nagios 80 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/tmp/var:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 nagiosramdisk
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/tmp/var/nagiosramdisk:
total 0
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 spool
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/tmp/var/nagiosramdisk/spool:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 .

/var/nagiosramdisk/spool:
total 0
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 perfdata
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 checkresults
drwxrwxrwt 4 nagios nagios 80 Nov 23 12:55 ..
drwxr-xr-x 4 nagios nagios 80 Nov 23 12:55 .

/var/nagiosramdisk/spool/perfdata:
total 0
drwxr-xr-x 4 nagios nagios 80 Nov 23 12:55 ..
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 .

/var/nagiosramdisk/spool/checkresults:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 var
drwxr-xr-x 4 nagios nagios 80 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/spool/checkresults/var:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 nagiosramdisk
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/spool/checkresults/var/nagiosramdisk:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 spool
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/spool/checkresults/var/nagiosramdisk/spool:
total 0
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 xidpe
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 .

/var/nagiosramdisk/spool/checkresults/var/nagiosramdisk/spool/xidpe:
total 0
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 ..
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 .

Looking directly at /var/nagiosramdisk/spool, I do not see an xidpe. however, I do see a /var/nagiosramdisk/spool/checkresults/var/nagiosramdisk/spool/xidpe folder

2:56 PM SERVER root [~]
$ cd /var/nagiosramdisk/spool/
12:56 PM SERVER root [/var/nagiosramdisk/spool]
$ ll
total 0
drwxr-xr-x 4 nagios nagios 80 Nov 23 12:55 .
drwxrwxrwt 4 nagios nagios 140 Nov 23 12:56 ..
drwxr-xr-x 3 nagios nagios 60 Nov 23 12:55 checkresults
drwxrwxr-x 2 nagios nagios 40 Nov 23 12:55 perfdata

Post by **pbroste** » Wed Nov 24, 2021 2:43 pm

Hello @hbouma

We see that the '/etc/init.d/nagios' and 'ramdisk.service' are calling for ramdisk start-up and want to disable the one found in /etc/init.d/nagios by commenting out following lines:

Please create a backup before editing:

Code: Select all

cp /etc/init.d/nagios /tmp/nagios.bak

Here are the lines that should be commented out with '#':

# Automate addition of RAMDISK based on environment variables
#USE_RAMDISK=${USE_RAMDISK:-0}
#if test "$USE_RAMDISK" -ne 0 && test "$RAMDISK_SIZE"X != "X"; then
# ramdisk=`mount |grep "${RAMDISK_DIR} type tmpfs"`
# if [ "$ramdisk"X == "X" ]; then
# if [ ! -d ${RAMDISK_DIR} ]; then
# mkdir -p -m 0755 ${RAMDISK_DIR}
# if [ -d ${RAMDISK_DIR} ]; then
# NagiosUserUID=`id -u ${NagiosUser}`
# NagiosGroupGID=`id -g ${NagiosGroup}`
# mount -t tmpfs -o uid=${NagiosUserUID},gid={$NagiosGroupGID},size=${RAMDISK_SIZE}m tmpfs ${RAMDISK_DIR}
# su ${NagiosUser} -c "mkdir -m 0755 ${RAMDISK_DIR}/checkresults"
# else
# echo "ERROR: Unable to create ramdisk directory (${RAMDISK_DIR})"
# exit 1
# fi
# fi
# fi
#fi

Attached is a copy just in case you want to drop it in and set permissions. Bounce the 'nagios.service' and 'ramdisk.service' and let us know how things look.

Thanks,
Perry

hbouma · Post by **hbouma** » Mon Nov 29, 2021 9:47 am

Our RHEL 7.9 server doesn't have a /etc/init.d/nagios file. We do have a /usr/lib/systemd/system/nagios.service file. However, the nagios.service file doesn't match the file you sent me.

Code: Select all

[Unit]
Description=Nagios Core 4.4.6
Documentation=https://www.nagios.org/documentation
After=network.target local-fs.target mariadb.service

[Service]
Type=forking
ExecStartPre=/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
ExecStart=/usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
ExecStop=/bin/kill -s TERM ${MAINPID}
ExecStopPost=/bin/rm -f /usr/local/nagios/var/rw/nagios.cmd
ExecReload=/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
ExecReload=/bin/kill -s HUP ${MAINPID}

[Install]
WantedBy=multi-user.target

Code: Select all

root [/etc/init.d]
$ ll
total 112K
drwxr-xr-x.  2 root root 4.0K Nov 16 07:56 .
drwxr-xr-x. 10 root root 4.0K Oct 13  2020 ..
-rwxr-xr-x   1 root root  16K Jun  1 12:34 Security Agent1
-rwxr-xr-x   1 root root 1.9K Nov  1 05:10 Security Agent 2
-rwxr-xr-x.  1 root root  22K Aug 21  2013 Security Agent 3
-rw-r--r--   1 root root  18K May 22  2020 functions
-rwxr-xr-x   1 root root 1.6K Feb 13  2021 ncpa_listener
-rwxr-xr-x   1 root root 1.5K Feb 13  2021 ncpa_passive
-rwxr-xr-x   1 root root 4.5K May 22  2020 netconsole
-rwxr-xr-x   1 root root 7.8K May 22  2020 network
-rwxr-xr-x   1 root root 2.1K Jan 20  2020 npcd
-rw-r--r--   1 root root 1.2K Jan 15  2021 README
-rwxrwxr-x.  1 root root 1.8K Nov  9  2011 rrdcached
-rw-rw-r--   1 root root 1.8K Nov  9  2011 rrdcached.1
lrwxrwxrwx.  1 root root   18 Mar 23  2017 Security Agent 4

Post by **pbroste** » Tue Nov 30, 2021 10:02 am

Hello @hbouma

Let's get an updated System Profile on this one.

To send us your system profile.

Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and send via Private Message

Thanks,
Perry

Post by **tgriep** » Tue Nov 30, 2021 10:05 am

It looks like there is a typo in the /usr/lib/systemd/system/ramdisk.service file. It is missing a space in the paths when it creates the folders.

Edit the /usr/lib/systemd/system/ramdisk.service file and make sure the ExecStartPre has a space between the /var/nagiosramdisk/spool/checkresults and the /var/nagiosramdisk/spool/xidpe when it creates the folders.

It should look like this.

Code: Select all

ExecStartPre=/usr/bin/mkdir -p -m 775 /var/nagiosramdisk/tmp/var/nagiosramdisk/spool /var/nagiosramdisk/spool/checkresults /var/nagiosramdisk/spool/xidpe /var/nagiosramdisk/spool/perfdata

Save the file and run the following to reload and restart the ramdisk service.

Code: Select all

systemctl daemon-reload
systemctl restart ramdisk

Then see if the /var/nagiosramdisk/spool/xidpe and the /var/nagiosramdisk/spool/checkresults folders are created and that the files are processed.

Thank you.

hbouma · Post by **hbouma** » Wed Dec 01, 2021 10:27 am

Fixing the ramdisk.service seems to have fixed the ramdisk filling up and appears to be processing properly. The performance graphs are also now showing up after I fixed the corrupt rrd files as listed at https://support.nagios.com/forum/viewto ... 20#p338587

Profile coming via PM.

Post by **pbroste** » Thu Dec 02, 2021 11:09 am

Hello @hbourma

Thanks for sending over the System Profile, we took a look through it and we are not seeing any issues that are sticking out.

Please let us know how things are looking and let us know where we need to direct our attention.

Thanks,
Perry

hbouma · Post by **hbouma** » Thu Dec 02, 2021 11:31 am

Everything seems to be working normally now. Go ahead and close this post.

Nagios Support Forum

Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs

Re: Nagios ramdisk full and no performance graphs