Nagios xi Monitor status "Error: Could not parse JSON from"

This support forum board is for support questions relating to Nagios xi, our flagship commercial network monitoring solution.
meganwilliford
Posts: 101
Joined: Tue Aug 06, 2019 7:49 am

Nagios xi Monitor status "Error: Could not parse JSON from"

Post by meganwilliford »

The past few weeks we've been intermittently getting a critical status of "Error: Could not parse JSON from..." for the Nagios xi monitors: I/O Wait, HTTP, Load, Nagios xi Jobs, and Nagios xi Daemon when monitoring other Nagios xi instances.

We've had these monitors running for over a year and just the past few weeks this started to exponentially occur where about every few runs we'll get "Error: Could not parse JSON from..." as the status. We have the same monitors set up on two other Nagios xi instances and this issue does not occur on those.

Any help on how to resolve this would be appreciated! Thanks!
Top
User avatar
vtrac
Posts: 903
Joined: Tue Oct 27, 2020 1:35 pm

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by vtrac »

Hi meganwilliford,
Since this is happening on and off, the server might be too busy at times to return json data fast enough.

Could you please check the following log locations for more detail errors?

Code: Select all

/var/log/httpd/
/usr/local/nagios/var/
/usr/local/nagiosxi/var/
/var/log/mariadb/mariadb.log
/var/log/messages
Also, please run the below two commands to check for "max_connections" and "Max_used_connections":

Code: Select all

mysql -uroot -pnagiosxi -e "show variables like 'max_connections';"

mysql -uroot -pnagiosxi -e "show global status like 'Max_used_connections';"
If the number returned is the same as (or close to max_connections) then you need to increase the allowed number of max_connections.
Please follow the URL below:
https://support.nagios.com/kb/article.php?id=513


Regards,
Vinh
Top
meganwilliford
Posts: 101
Joined: Tue Aug 06, 2019 7:49 am

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by meganwilliford »

Hi Vinh,

This is still occurring. Is there anything in particular to look for in the logs? The max_used_connections we're a little over 100 which I didn't think was close to the max_connections of 151 but I increased it anyway to see if that would help and so far no. Any other ideas?

Thanks!
Top
User avatar
vtrac
Posts: 903
Joined: Tue Oct 27, 2020 1:35 pm

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by vtrac »

Hi meganwilliford,
Please check your system and make sure that you don't have something BIG that is running which will cause the parsing of the JSON to time out.

I would first restart (reboot) your VM, then wait for about 30 minutes and see how things goes.
I have seen in the past where there were many jobs (processes) that were stuck and rebooting the machine does helps.

To further investigate the issue, could you please send me the "profile.zip"
and the exact name of the host and services sending notifications.

To send us your system profile.
Login to the Nagios xi GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to the post/ticket

Additionally, please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table
Regards,
Vinh
Top
meganwilliford
Posts: 101
Joined: Tue Aug 06, 2019 7:49 am

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by meganwilliford »

PM sent!
Top
User avatar
vtrac
Posts: 903
Joined: Tue Oct 27, 2020 1:35 pm

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by vtrac »

Hi,
Please also send the output of these commands (as root):

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

Regards,
Vinh
Top
meganwilliford
Posts: 101
Joined: Tue Aug 06, 2019 7:49 am

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by meganwilliford »

Sent that in a PM also. Let me know if you need anything else.
Top
User avatar
vtrac
Posts: 903
Joined: Tue Oct 27, 2020 1:35 pm

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by vtrac »

Hi,
Looking at the profile.zip, I found two things that need attention.

1) You have a total of "670" running processes.
You MUST reboot your machine to clear things out. This might cause your JSON to time out or failed.
Please REBOOT your VM (machine)!! ..... I don't mean restart Nagios .... I meant reboot the VM.
I have seen JSON issue in the past with hundred of running processes and reboot cleared things up.


2). You have the below errors, which will cause your Nagios to NOT being able to start or restart.
Error: Could not find any hostgroup matching 'windows-servers' (config file '/usr/local/nagios/etc/hosts/ctx-xxx.cfg', starting on line 16)
Error: Failed to process hostgroup names for host 'ctx-xxx' (config file '/usr/local/nagios/etc/hosts/ctx-xxx.cfg', starting at line 16)


ctx-xxx.cfg: As you can see 'windows-servers' hostgroup is defined .... must be remove using CCM, do not edit the cfg file directly.

Code: Select all

define host {
    host_name                ctx-xxx
    use                      xiwizard_ncpa_host
    address                  ctx-xxx
    hostgroups               Citrix Production,Windows Servers
    check_command            check_xi_service_http!-s "Success" -f ok -u '/testconnect/?token=b0eing_ncpa' -S --sni -p 5693!!!!!!!
    max_check_attempts       10
    check_interval           5
    retry_interval           1
    check_period             xi_timeperiod_24x7
    contact_groups           Citrix
    notification_interval    60
    notification_period      xi_timeperiod_24x7
    notifications_enabled    0
    icon_image               win_server.png
    statusmap_image          win_server.png
    _xiwizard                ncpa
    register                 1
}
To remove hostgroups 'windows-servers' off of "ctx-xxx", please use CCM (do not edit the file directly):
Nagios xi GUI > Configure > Core Config Manager > Host
Click "ctx-xxx" > click "Mange Host Groups" ..... remove "windows-servers" off of the list on the "Assigned".
Apply Configuration


3). Your database log said it crashed, please run the "repair_databases.sh" script:
[ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Also, your Nagios's log was empty (missing), so I can not see what is happening to Nagios.


Regards,
Vinh
Top
meganwilliford
Posts: 101
Joined: Tue Aug 06, 2019 7:49 am

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by meganwilliford »

PM sent. And, thank you for the feedback! I will work these steps and then monitor the status after.
Top
User avatar
vtrac
Posts: 903
Joined: Tue Oct 27, 2020 1:35 pm

Re: Nagios xi Monitor status "Error: Could not parse JSON fr

Post by vtrac »

Hi,
As requested, I have removed the server name off of my last reply for security reason.

Regards,
Vinh
Top

Return to “Nagios xi”