Monitoring Event Queue is not showing data in GUI.

Bwick05 · Post by **Bwick05** » Fri Jul 09, 2021 6:06 pm

We are encountering a similar behavior as what's outlined here (https://support.nagios.com/forum/viewto ... 16&t=60863).

We upgraded from 5.6.13 to 5.8.3 in our non-PROD environment and are having performance display issues. We see data is in the check Performance graph tabs, but not in the check Overview tabs.

We are waiting to commit to PROD until this issue is fixed/Root Caused as there is mention of a known bug in 5.7+, but the thread is locked with no fix called out.

Please assist.

ssax · Post by **ssax** » Mon Jul 12, 2021 9:50 am

You could be hitting a bug if you're running XI 5.7+ and running the new NDO3.

Please PM me a copy of your profile.zip, you can download it from Admin > System Profile by clicking the Download Profile button.

Attach this file as well:

Code: Select all

/usr/local/nagios/var/nagios.log

Additionally, please send the output of these commands:
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

code]
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table[/code]

This next command may fail, that's okay, not all systems run postgresql, send the output anyways:

Code: Select all

echo "SELECT relname as Table, pg_size_pretty(pg_total_relation_size(relid)) As Size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as ExternalSize FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;" | psql nagiosxi nagiosxi

ssax · Post by **ssax** » Thu Jul 15, 2021 5:31 pm

Please go to Admin > Performance Settings > PDF/JPG Exporting and uncheck (or check, whichever is the opposite of what it currently is) and then click the Update Settings button. Then check the graphs/event queue again.

Follow this guide and increase your load_threshold to 60.0 and your TIMEOUT to 20:

https://support.nagios.com/kb/article.p ... ategory=44

You only have 4GB of RAM, I recommend that you add more as you're swapping and increase your /etc/php.ini memory_limit to 1024M and restart apache.

Bwick05 · Post by **Bwick05** » Fri Jul 16, 2021 10:55 am

ssax wrote:
Follow this guide and increase your load_threshold to 60.0 and your TIMEOUT to 20:

https://support.nagios.com/kb/article.p ... ategory=44

Apologies, but I am not understanding, the link you provided is with regards to sudoers, I am not seeing how to adjust the load_threshold and timeout as you describe. Can you assist? Thanks in advance!

gsmith · Post by **gsmith** » Fri Jul 16, 2021 2:50 pm

Hi,

Please use this: https://support.nagios.com/kb/pdf-9.html

Thanks

Bwick05 · Post by **Bwick05** » Tue Sep 07, 2021 3:41 pm

I have tried all that is suggested with no resolution. I have upgraded to 5.8.4 in hopes the upgrade would fix the issue, that did not work. I have reinstalled the ndoutils module, that resolved 1 portion of my issue. Our non-prod environment is still broke after upgrading, and this is piling up vulnerabilities in our PROD environment because we cannot move forward until this issue is resolved. Please help, the business is losing confidence in this platform and is taking on risk. Please let me know what else needs to be done to resolve.

Another issue now shows up in our nagios.log files

[1631045157] NDO-3: ndo_handle_service_status(ndo-handlers.c:1096): Unable to bind parameters
[1631045158] NDO-3: ndo_return = 1 (Statement not prepared)

and

[1631045130] NDO-3: ndo_handle_host_status(ndo-handlers.c:985): Unable to bind parameters
[1631045130] NDO-3: ndo_return = 1 (Statement not prepared)

Lastly, archives have a gap in them starting approximately when the issue was first encountered. And resumes once we commented out the broker_module.

-rw------- 1 nagios nagios 85544 Jun 7 23:17 nagios-06-08-2021-00.log
-rw------- 1 nagios nagios 89362 Jun 8 23:58 nagios-06-09-2021-00.log
-rw------- 1 nagios nagios 82414 Jun 9 23:49 nagios-06-10-2021-00.log
-rw------- 1 nagios nagios 136430 Jun 10 23:59 nagios-06-11-2021-00.log
-rw------- 1 nagios nagios 79548 Jun 11 23:17 nagios-06-12-2021-00.log
-rw------- 1 nagios nagios 80479 Jun 12 23:52 nagios-06-13-2021-00.log
-rw------- 1 nagios nagios 93232 Jun 13 00:00 nagios-06-07-2021-00.log
-rw------- 1 nagios nagios 94183 Jul 10 23:44 nagios-07-11-2021-00.log
-rw------- 1 nagios nagios 104536 Jul 11 00:00 nagios-07-10-2021-00.log
-rw------- 1 nagios nagios 104232 Jul 12 23:00 nagios-07-13-2021-00.log
-rw------- 1 nagios nagios 263390 Jul 13 23:56 nagios-07-14-2021-00.log
-rw------- 1 nagios nagios 91963 Jul 14 23:00 nagios-07-15-2021-00.log
-rw------- 1 nagios nagios 305270 Jul 15 23:00 nagios-07-16-2021-00.log

gsmith · Post by **gsmith** » Tue Sep 07, 2021 4:21 pm

Hi,

On your non-prod system please provide a System Profile
To send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share this in a private message and then reply to this post to bring it up in the queue.

Also please provide the output of this on your non-prod system:

Code: Select all

mysql -uroot -pnagiosxi nagios -e 'show create table nagios_hoststatus;  show create table nagios_servicestatus'

On your non-prod system please go to the following URL and let us know if you see something obviously wrong:
http://<YOURXISERVER>/nagios/

Thanks

gsmith · Post by **gsmith** » Thu Sep 09, 2021 1:51 pm

Hi,

I am seeing a ton of:

Code: Select all

00166: [Sun Sep 05 07:50:33.386908 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58572] script not found or unable to stat: /var/www/cgi-bin/ecrire
00167: [Sun Sep 05 07:50:33.460526 2021] [authz_core:error] [pid 27169] [client 10.41.51.45:57992] AH01630: client denied by server configuration: /var/www/mrtg/includes
00168: [Sun Sep 05 07:50:33.503536 2021] [cgi:error] [pid 28985] [client 10.41.51.45:58637] script not found or unable to stat: /var/www/cgi-bin/plugins
00169: [Sun Sep 05 07:50:33.566048 2021] [authz_core:error] [pid 24196] [client 10.41.51.45:58674] AH01630: client denied by server configuration: /var/www/mrtg/default
00170: [Sun Sep 05 07:50:33.612169 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58710] script not found or unable to stat: /var/www/cgi-bin/plugins
00171: [Sun Sep 05 07:50:33.674608 2021] [authz_core:error] [pid 27169] [client 10.41.51.45:58744] AH01630: client denied by server configuration: /var/www/mrtg/atk
00172: [Sun Sep 05 07:50:33.718336 2021] [cgi:error] [pid 28985] [client 10.41.51.45:58764] script not found or unable to stat: /var/www/cgi-bin/plugins
00173: [Sun Sep 05 07:50:33.781480 2021] [authz_core:error] [pid 24196] [client 10.41.51.45:58810] AH01630: client denied by server configuration: /var/www/mrtg/atk
00174: [Sun Sep 05 07:50:33.824980 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58838] script not found or unable to stat: /var/www/cgi-bin/plugins
00175: [Sun Sep 05 07:50:33.888592 2021] [authz_core:error] [pid 30129] [client 10.41.51.45:58880] AH01630: client denied by server configuration: /var/www/mrtg/atk
00176: [Sun Sep 05 07:50:33.931570 2021] [cgi:error] [pid 85854] [client 10.41.51.45:58916] script not found or unable to stat: /var/www/cgi-bin/plugins

There were changes in authorization for Apache going from 2.2 to 2.4 as shown in:
http://httpd.apache.org/docs/2.4/upgrad ... l#run-time

Please verify you are on Apache 2.4:
run: httpd -v

If you are on 2.4 look at your conf files under /etc/httpd/conf.d. The changes in the link referenced above boil down to:

Remove any “Order deny,allow”, “Order allow,deny”, and related lines
Replace “Deny from all” to “Require all denied”
Replace “Allow from all” to “Require all granted”
If you are allowing any specific hostname, change them accordingly. For example, replace “Allow from 192.168.1.2” to “Require host 192.168.1.2”

Verify that the conf files have been updated to reflect the above. If you have to make changes please preserve the conf files
by making a copy/renaming them prior to making changes.

Let me know what you find out.

Thanks

Bwick05 · Post by **Bwick05** » Thu Sep 09, 2021 3:38 pm

gsmith wrote:Hi,

I am seeing a ton of:
Code: Select all
00166: [Sun Sep 05 07:50:33.386908 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58572] script not found or unable to stat: /var/www/cgi-bin/ecrire
00167: [Sun Sep 05 07:50:33.460526 2021] [authz_core:error] [pid 27169] [client 10.41.51.45:57992] AH01630: client denied by server configuration: /var/www/mrtg/includes
00168: [Sun Sep 05 07:50:33.503536 2021] [cgi:error] [pid 28985] [client 10.41.51.45:58637] script not found or unable to stat: /var/www/cgi-bin/plugins
00169: [Sun Sep 05 07:50:33.566048 2021] [authz_core:error] [pid 24196] [client 10.41.51.45:58674] AH01630: client denied by server configuration: /var/www/mrtg/default
00170: [Sun Sep 05 07:50:33.612169 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58710] script not found or unable to stat: /var/www/cgi-bin/plugins
00171: [Sun Sep 05 07:50:33.674608 2021] [authz_core:error] [pid 27169] [client 10.41.51.45:58744] AH01630: client denied by server configuration: /var/www/mrtg/atk
00172: [Sun Sep 05 07:50:33.718336 2021] [cgi:error] [pid 28985] [client 10.41.51.45:58764] script not found or unable to stat: /var/www/cgi-bin/plugins
00173: [Sun Sep 05 07:50:33.781480 2021] [authz_core:error] [pid 24196] [client 10.41.51.45:58810] AH01630: client denied by server configuration: /var/www/mrtg/atk
00174: [Sun Sep 05 07:50:33.824980 2021] [cgi:error] [pid 85852] [client 10.41.51.45:58838] script not found or unable to stat: /var/www/cgi-bin/plugins
00175: [Sun Sep 05 07:50:33.888592 2021] [authz_core:error] [pid 30129] [client 10.41.51.45:58880] AH01630: client denied by server configuration: /var/www/mrtg/atk
00176: [Sun Sep 05 07:50:33.931570 2021] [cgi:error] [pid 85854] [client 10.41.51.45:58916] script not found or unable to stat: /var/www/cgi-bin/plugins
There were changes in authorization for Apache going from 2.2 to 2.4 as shown in:
http://httpd.apache.org/docs/2.4/upgrad ... l#run-time

Please verify you are on Apache 2.4:
run: httpd -v

If you are on 2.4 look at your conf files under /etc/httpd/conf.d. The changes in the link referenced above boil down to:
Remove any “Order deny,allow”, “Order allow,deny”, and related lines
Replace “Deny from all” to “Require all denied”
Replace “Allow from all” to “Require all granted”
If you are allowing any specific hostname, change them accordingly. For example, replace “Allow from 192.168.1.2” to “Require host 192.168.1.2”
Verify that the conf files have been updated to reflect the above. If you have to make changes please preserve the conf files
by making a copy/renaming them prior to making changes.

Let me know what you find out.

Thanks

I have checked both our PROD (5.6.13) and non-PROD (5.8.4), both are running apache in version 2.4.

[user@server ~]$ httpd -v
Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Server built: Oct 8 2020 21:27:40
[user@server ~]$

I also verified the paths called out in the error messages you posted and there are no files located at either server (PROD or non-PROD).

[user@server ~]$ sudo ls /var/www/mrtg/atk
ls: cannot access /var/www/mrtg/atk: No such file or directory
[user@server ~]$

Same result for "/var/www/cgi-bin/plugins"

Can you assist with what specific config files need to be updated? The /etc/httpd/conf.d folder contains many configs and I fear changing configurations in non-PROD configs manually knowing the same config and apache version state exists in PROD (and PROD works as expected), adds a level of risk.

Does the information attached helpful to pinpoint a root cause? Also is this an issue that should be worked via a support call or something more formal?

Thanks!

gsmith · Post by **gsmith** » Fri Sep 10, 2021 9:09 am

Hi

I am continuing to look through the logs for more errors and am going to pull in a colleague.

In the meantime can you zip up /etc/httpd/conf.d directory and I will take a look at the config files.

Thanks

Nagios Support Forum

Monitoring Event Queue is not showing data in GUI.

Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.

Re: Monitoring Event Queue is not showing data in GUI.