Good morning Nagios team,
This morning I experienced what I would consider to be a "typical" crash/down scenario for my environment. Essentially, I logged into my Log Server console just after 7AM PST when I started my shift and everything was fine for a few hours in between periodic refreshes of the 'Home' screen. We're troubleshooting another issue in a separate support ticket, but basically what will happen is the my session will timeout in the web UI and show a yellow triangle in the upper right-hand corner and the very next thing I click on once that has happened will completely log me out of the current session.
Well, I went to log back in as I typically would (around 10:45AM PST) and the console became unresponsive, just a spinning circle on that tab in Chrome. I closed the tab and tried logging in again to no avail. I checked my vSphere console for any strange CPU behavior and noticed that on one of my nodes the CPU activity was wildly fluctuating every 30 seconds or so. I made the decision to reboot that node and after a few minutes once all the CPU activity normalized I tested logging into the console and it worked. The Home graph shows that logstash collection crashed/stopped just before 8AM PST, but when I SSH'd into all three of the nodes they all showed both logstash and elasticsearch running (active).
Whenever I experience an issue with Log Server, this is the behavior I see 95% of the time. I don't understand why the logstash service shows it's running when it very clearly isn't. Can I send a system profile to someone for review? Maybe you can see what's happening?
Thank you.
Logstash crash and unable to access web UI
-
- Posts: 733
- Joined: Wed Jul 11, 2018 11:37 am
-
- Support Tech
- Posts: 5045
- Joined: Tue Feb 07, 2017 11:26 am
Re: Logstash crash and unable to access web UI
Yes, please PM me a profile.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 733
- Joined: Wed Jul 11, 2018 11:37 am
Re: Logstash crash and unable to access web UI
This happened again today, if you could let me know whether or not you see anything odd in the system profiles I sent you, I'd appreciate it.
-
- Support Tech
- Posts: 5045
- Joined: Tue Feb 07, 2017 11:26 am
Re: Logstash crash and unable to access web UI
Responded in the PM.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 733
- Joined: Wed Jul 11, 2018 11:37 am
Re: Logstash crash and unable to access web UI
Yes, we have numerous people that are able to login to the console and perform searches.
Is it possible to change the default dashboard for every account in my environment? I think everyone is using the default query (* + 24Hrs) at this time. I would very much like to change that to the empty dashboard for everyone.
Thank you!
Is it possible to change the default dashboard for every account in my environment? I think everyone is using the default query (* + 24Hrs) at this time. I would very much like to change that to the empty dashboard for everyone.
Thank you!
-
- Support Tech
- Posts: 5045
- Joined: Tue Feb 07, 2017 11:26 am
Re: Logstash crash and unable to access web UI
While it is possible do to from the command line I would recommend advising the users to make the change through the web UI.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 733
- Joined: Wed Jul 11, 2018 11:37 am
Re: Logstash crash and unable to access web UI
For some reason I cannot reply to your last PM, the message is just sitting in the Outbox.
I didn't get the attachment you tried to send, there's nothing there. Thank you.
I didn't get the attachment you tried to send, there's nothing there. Thank you.
-
- Support Tech
- Posts: 5045
- Joined: Tue Feb 07, 2017 11:26 am
Re: Logstash crash and unable to access web UI
Resent.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 733
- Joined: Wed Jul 11, 2018 11:37 am
Re: Logstash crash and unable to access web UI
This thread can be locked. The issue is resolved. Thank you.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
Re: Logstash crash and unable to access web UI
Great!rferebee wrote:This thread can be locked. The issue is resolved. Thank you.
Locking thread