Nagios XI exploit

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
MarkCampbell
Posts: 20
Joined: Tue Oct 16, 2018 4:29 pm

Nagios XI exploit

Post by MarkCampbell »

hello,

My AWS hosted Nagios server has been attacked a few times now, and I'm not quite sure how to prevent it. I was running 5.7.4 for the last few months (I just now upgraded to 5.8.1). The behavior I'm seeing resembles CVE-2020-15901, but that was supposed to have been closed in 5.7.2. Here's a sample of the log entries:

Code: Select all

/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_servicestatus_table%22%2C%22args%22%3A%7B%22host%22%3A%22%22%2C%22hostgroup%22%3A%22%22%2C%22servicegroup%22%3A%22%22%2C%22sortby%22%3A%22%22%2C%22sortorder%22%3A%22asc%22%2C%22records%22%3A%221000%22%2C%22page%22%3A%221%22%2C%22search%22%3A%22%22%2C%22hostattr%22%3A%220%22%2C%22serviceattr%22%3A%220%22%2C%22hoststatustypes%22%3A%220%22%2C%22servicestatustypes%22%3A%2228%22%7D%7D&nsp=63196d87b01311f13c3c8b87ca2385b04033edfe59b2f44fa97b6497f0db63d4
/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D&nsp=63196d87b01311f13c3c8b87ca2385b04033edfe59b2f44fa97b6497f0db63d4
The attacker manages to kick off a wget of https://sslcer.justdied.com:8080/watch.sh, and executes it, but all only as the apache user, so a number of lines in the script fail, since it assumes root access. But it does download a binary called avalonsaber into /tmp, and executes as apache user, which pegs the CPU (which is how I discovered the attack).

I set a number of AWS WAF managed rules that are supposed to help manage common exploits, etc, including rate limiting rules. But most of them seem to still make it through as allowed. I also had the nagiosadmin using insecure tickets, as it appeared to be the only way I could get nagios license checks going, but I've now disabled it, as I fear that's how they're gaining authenticated access.

Any suggestions on how to 1) prevent these attacks if I'm already at a version that was supposed to have this closed? 2) prevent the apache user from executing wget / curl commands? 3) Any AWS WAF rules I can add in to help prevent this behavior moving forward? 4) Alternative methods of checking the license expiration in Nagios?
Top
pspagnola
Posts: 51
Joined: Thu Nov 21, 2019 4:40 pm

Re: Nagios XI exploit

Post by pspagnola »

I am not an AWS expert or a Nagios expert, but I do have some experience with protecting web servers from injection attacks.

First, I hope that Nagios support can give you a patch to protect from this. The fact that they can upload a binary file to your /tmp is very scary!

As far as general security concepts go and what can be applied to help you, here are some thoughts I have:
  • - Do you use that specific URL? Can you setup the AWS to block that URL from external access?
    - If you need that URL open for certain addresses, can you build a group in AWS firewall that contains the addresses that need access?
    - If your devices or users don't have static addresses, perhaps there is an AWS software VPN client that you can use to access the Nagios XI interface and disable all non-VPN access for external users.
    - Is there any pattern in the attacker's IP address? perhaps you can block their IP range or country?
    - AWS help pages indicate that you can filter on various things. Perhaps you can block requests by size or request length or user agent or something like that. Reviewing the log entries related to the attack are key to finding the right combination of things to block.
    - Maybe you should block outbound internet requests form your Nagios server to unapproved sites. This would make it hard for the server to download a payload from an attacker controlled webpage.
    - If you don't have a need to access "get_admin_tasks_html" like that request does, perhaps you can block that via AWS WAF regex pattern. (Assuming that is a crucial step in the attack, I don't have enough info to tell)
Nagios patches are always going to be a temporary fix until the next bug is found. Finding a way to reduce your attack surface by limiting who can access your Nagios server (or at least certain services and URLs) will have a more lasting effect.

I suspect your logs may have other relevant entries for this attack. That URL when decoded does not appear to contain any malicious payload.
Nagios v5.6.12 ~200 hosts 500+ services.
Top
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Nagios XI exploit

Post by dchurch »

This is a known issue.

Ensure the Docker config Wizard is at least version 1.1.2 to prevent further infection. Instructions for how to do that can be found at this link: https://support.nagios.com/kb/article/n ... s-836.html

You can update the Docker wizard without having to upgrade Nagios XI. When you installed 5.8.1, it should have pulled in all available wizard updates including the Docker wizard update.

You will ALSO then want to run this script as the root user to disinfect the machine:

Code: Select all

for (( x = 0; x < 100; ++x)); do
chattr -i /etc/crontab
chattr -i /tmp/avalonsaber
chattr -R -i /var/spool/cron
echo | crontab -
echo | crontab -u nagios -

pkill -9 avalonsaber
pkill -9 lwp-download
pkill -9 curl
pkill -9 wget

rm -rf /tmp/avalonsaber
done
Security disclosures can be found at this page: https://www.nagios.com/products/security/
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Top
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Nagios XI exploit

Post by dchurch »

Unfortunately it's not as simple as blocking requests to specific URL's, or prevent shell from executing; some web scripts are designed to let the user execute shell scripts.

This may have been preventable with something like SELinux that might have prevented Apache from modifying a crontab (if SELinux is configured correctly). Sadly, SELinux is not the easiest to work with, or I'd recommend it for the security-concerned.

Another server-hardening route I've seen customers take is requiring VPN access to get to the Nagios XI server page.

It's an old trick, but you could also put the Apache server to run on a non-standard port that might not show up in scans.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Top
MarkCampbell
Posts: 20
Joined: Tue Oct 16, 2018 4:29 pm

Re: Nagios XI exploit

Post by MarkCampbell »

Hi, thanks for the quick replies! some quick responses:
pspagnola:
-Unfortunately, the site needs to remain public (ease of access from mobile devices that receive alerts), and the logs show the attacker trying all of the pages, so blocking some of them may not work.
-There were 2 attacks. The first one happened from a single IP in China, doing 1000 hits/5 min. So I added some rules, blocking that IP, and adding rate limiting (100 hits/5min is AWS minimum). Second attack came from a US IP address, and they adjusted their rate down to 80 hits/5 min. I've blocked that IP too.
-I am considering limiting outbound traffic, right now my AWS security groups restrict incoming pretty heavily, but outbound is fairly open.

dchurch:
-I've confirmed that the docker config wizard (and everything else) was updated, probably with my 5.8.1 upgrade.
-Thanks for a recommended list of commands to run. I had already ran a few of them, deleting avalonsaber, and removing entries from apache's crontab. I did check the rest of the commands, and since the script ran as apache, it didn't have root permissions, and so none of those other commands were showing anything out of the ordinary.
-I thought that Nagios required SELinux to be disabled, either as a prerequisite for install, or it just did it as part of the install? I've been in the linux world long enough to remember a time before selinux was part of centos, at least. And I'm sometimes guilty of still following the old days' practice of just disabling selinux as part of my first command on a new system, just because it caused more problems than it solved back then.
-I unfortunately can't do VPN or alternative ports, as they would interfere with the execs' ability to check in on their mobile devices; plus, depending on the problem, our VPN could potentially be knocked out, preventing us from being able to see the alerts at all. We decided to have it open so that it would be the least likely to be affected by system outages where it is.
Top
connected

Re: Nagios XI exploit

Post by connected »

Running any application publicly without decent security device/service in front is a very bad idea.
Just imagine that they obtain root access. Just brute forcing your user credentials (or using the one found in a 3rd party data breech).
There probably are plenty of credentials in the Nagios checks to totally wipe out your company.

Even just a simple proxy in front that requires a simple password will be a lot safer than being totally open.
There are mobile VPN clients too!
I've even seen self-build solutions where the mobile IP first needs to be added to an allow list which was editable via a simple public webapplication with two-factor authentication.
Top
MarkCampbell
Posts: 20
Joined: Tue Oct 16, 2018 4:29 pm

Re: Nagios XI exploit

Post by MarkCampbell »

connected:
Let me be clear, since you apparently missed some of what I said before, it's not completely open. It is in AWS, sitting behind a load balancer, which has AWS's WAF enabled. There's 1,400 managed rules in place to block bad actors, which does catch plenty of stuff. It just unfortunately didn't catch this particular attack, which seems to be a problem specific to Nagios, and doesn't show up on the standard set of rules.

We require a username/password to log in (which is standard with Nagios XI), and the logs don't show any indications of a user/pass breach.

There's no need to be snarky. I am well aware of mobile VPN clients existing, but having access to this as you would any other password protected web application in the cloud (such as Office 365) was a high priority from the top. At any rate, this is not central to my remaining questions (#3 & #4). If you have answers to any of those, I'd welcome them.
Top
dchurch
Posts: 858
Joined: Wed Oct 07, 2020 12:46 pm
Location: Yo mama

Re: Nagios XI exploit

Post by dchurch »

MarkCampbell wrote:We require a username/password to log in (which is standard with Nagios XI), and the logs don't show any indications of a user/pass breach.
That follows with the nature of the exploit; it was an attack on a file that one didn't need to be logged in in order to execute.

Let us know if you're having any other issues with this or other exploits.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.

Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Top
MarkCampbell
Posts: 20
Joined: Tue Oct 16, 2018 4:29 pm

Re: Nagios XI exploit

Post by MarkCampbell »

Actually, I think I do still have an issue. As I mentioned before, I have my Nagios server behind an AWS WAF, and I discovered that there were still some attacks going on, most of them failing to get past the WAF. However, when I started up for work this morning, I discovered that there was an attack going on from my home IP address (I work from home). So I did some digging, and found that it was definitely my work laptop, and that it appears to be a server-side javascript attack, as I only had a couple of tabs with nagios open, but despite that they were logged out (timed out), I was getting some 40-90 hits per 5 min according to the WAF logs. See a sample of the logs here (IPs masked):

Code: Select all

Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_getting_started_html%22%2C%22args%22%3A%22%22%7D&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:04:55 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/includes/js/dashlets.js?1610744847	-	ALLOW	Mon Jan 25 2021 13:22:35 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/includes/dashlets/textenhanced/textenhanced.js	-	ALLOW	Mon Jan 25 2021 13:22:36 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:04:55 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/includes/js/views.js?1610744847	-	ALLOW	Mon Jan 25 2021 13:22:35 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=keepalive&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:03:55 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=keepalive&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:05:55 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:08:55 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:11:25 GMT-0500 (Eastern Standard Time)
Nagios-acl	xxx.xxx.xxx.xxx (US)	/nagiosxi/ajaxhelper.php?cmd=keepalive&nsp=205d3ad988e9eb0d7a8207c9a79f13027404427d9b92196a1575703d2d50d480	-	ALLOW	Mon Jan 25 2021 13:04:55 GMT-0500 (Eastern Standard Time)
I browsed to the login.php page again, and used developer tools to take a look at the code, and I saw this at the beginning (full url replaced with example.org):

Code: Select all

<script type="text/javascript">
    var base_url = "https://nagios.example.org/nagiosxi/";
    var backend_url = "https%3A%2F%2Fnagios.example.org%2Fnagiosxi%2Flogin.php";
    var ajax_helper_url = "https://nagios.example.org/nagiosxi/ajaxhelper.php";
    var ajax_proxy_url = "https://nagios.example.org/nagiosxi/ajaxproxy.php";
    var suggest_url = "https://nagios.example.org/nagiosxi/suggest.php";
    var request_uri = "%2Fnagiosxi%2Flogin.php%3Fredirect%3D%2Fnagiosxi%2Findex.php%253f%26noauth%3D1";
    var demo_mode = 0;
    var nsp_str = "3a8821ad1494dd3c6ea60e8c63fbd36fb6be534f2cf163b4d8689cf332cb601b";
    var theme = "xi5";

    // Language string for translations
    var lang = {
        'Add to Dashboard': "Add to Dashboard",
        'Add Dashboard': "Add Dashboard",
        'Edit Dashboard': "Edit Dashboard",
        'Dashlet Title': "Dashlet Title",
        'Dashboard Added': "Dashboard Added",
        'Add It': "Add It",
        'Add this powerful little dashlet to one of your dashboards for visual goodness.': "Add this powerful little dashlet to one of your dashboards for visual goodness.",
        'Select a Dashboard to Add To': "Select a Dashboard to Add To",
        'Add this graph to a dashboard.': "Add this graph to a dashboard.",
        'Dashlet is now loaded on your dashboard.': "Dashlet is now loaded on your dashboard.",
        'Dashlet Added': "Dashlet Added",
        'Please Wait': "Please Wait",
        'Submitting command': "Submitting command",
        'Show Details': "Show Details",
        'Hide Details': "Hide Details",
        'Show password': "Show password",
        'Hide password': "Hide password",
        'Permalink': "Permalink",
        'Copy the URL below to retain a direct link to your current view.': "Copy the URL below to retain a direct link to your current view.",
        'URL': "URL",
        'Thank You!': "Thank You!",
        'Thanks for helping to make this product better! We will review your comments as soon as we get a chance. Until then, kudos to you for being awesome and helping drive innovation!': "Thanks for helping to make this product better! We will review your comments as soon as we get a chance. Until then, kudos to you for being awesome and helping drive innovation!",
        'Error': "Error",
        'An error occurred. Please try again later.': "An error occurred. Please try again later.",
        'Sending Feedback': "Sending Feedback",
        'Use this to add a new dashboard to your Dashboards page.': "Use this to add a new dashboard to your Dashboards page.",
        'Dashboard Title': "Dashboard Title",
        'Background Color': "Background Color",
        'Submit': "Submit",
        'Processing': "Processing",
        'Success! Your new dashboard has been added.': "Success! Your new dashboard has been added.",
        'An error occurred processing your request.': "An error occurred processing your request.",
        'Dashboard Changes Saved': "Dashboard Changes Saved",
        'Success! Your dashboard was updated successfully.': "Success! Your dashboard was updated successfully.",
        'You cannot delete your home page dashboard.': "You cannot delete your home page dashboard.",
        'Confirm Dashboard Deletion': "Confirm Dashboard Deletion",
        'Are you sure you want to delete this dashboard and all dashlets it contains?': "Are you sure you want to delete this dashboard and all dashlets it contains?",
        'Delete': "Delete",
        'Cancel': "Cancel",
        'The requested dashboard has been deleted.': "The requested dashboard has been deleted.",
        'Dashboard Deleted': "Dashboard Deleted",
        'Clone Dashboard': "Clone Dashboard",
        'Use this to make an exact clone of the current dashboard and all its wonderful dashlets.': "Use this to make an exact clone of the current dashboard and all its wonderful dashlets.",
        'Clone': "Clone",
        'New Title': "New Title",
        'Dashboard Cloned': "Dashboard Cloned",
        'Dashboard successfully cloned.': "Dashboard successfully cloned.",
        'Deleting dashlets from the home page dashboard is disabled while in demo mode.': "Deleting dashlets from the home page dashboard is disabled while in demo mode.",
        'Dashlet Deleted': "Dashlet Deleted",
        'Dashlet removed from dashboard.': "Dashlet removed from dashboard.",
        'The dashlet has been added and will now show up on your dashboard.': "The dashlet has been added and will now show up on your dashboard.",
        'Masquerade Notice': "Masquerade Notice",
        'You are about to masquerade as another user. If you choose to continue you will be logged out of your current account and logged in as the selected user. In the process of doing so, you may lose your admin privileges.': "You are about to masquerade as another user. If you choose to continue you will be logged out of your current account and logged in as the selected user. In the process of doing so, you may lose your admin privileges.",
        'Continue': "Continue",
        'Add View': "Add View",
        'Use this to add what you see on the screen to your views page.': "Use this to add what you see on the screen to your views page.",
        'View Title': "View Title",
        'View Added': "View Added",
        'Success! Your view was added to your views page.': "Success! Your view was added to your views page.",
        'View Deleted': "View Deleted",
        'View has been removed.': "View has been removed.",
        'Edit View': "Edit View",
        'View Changes Saved': "View Changes Saved",
        'Success! Your view was updated successfully.': "Success! Your view was updated successfully.",
        'Start Rotation': "Start Rotation",
        'Stop Rotation': "Stop Rotation",
        'Pause rotation': "Pause rotation",
        'Resume rotation': "Resume rotation",
        'You are about to delete the view': "You are about to delete the view",
        'Cannot schedule outside pages.': "Cannot schedule outside pages.",
        'Any page not under nagiosxi cannot be scheduled.': "Any page not under nagiosxi cannot be scheduled.",
        'Loading': "Loading",
        'Update': "Update",
        'Close': "Close",
        'Time Range': "Time Range",
        'Last 4 Hours': "Last 4 Hours",
        'Last 24 Hours': "Last 24 Hours",
        'Last Week': "Last Week",
        'Last Month': "Last Month",
        'Last Year': "Last Year",
        'Last 7 Days': "Last 7 Days",
        'Last 30 Days': "Last 30 Days",
        'Last 365 Days': "Last 365 Days",
        'My graph': "My graph",
        'You must fill out the entire form.': "You must fill out the entire form.",
        'Copy to Clipboard': "Copy to Clipboard",
        'Copied': "Copied",
        'Press Ctrl+C to copy': "Press Ctrl+C to copy",
        'Dismiss' : "Dismiss"
    };

    // Translation helper function
    function _(str) {
        var trans = lang[str];
        if (trans) { return trans; }
        return str;
    }
    </script>
What can I do to clear this up? the timestamps of the login.php files suggests that they haven't been modified since I upgraded to 5.8.1, which is troubling since that suggests that this code might have survived an upgrade. Any suggestions?
Top
MarkCampbell
Posts: 20
Joined: Tue Oct 16, 2018 4:29 pm

Re: Nagios XI exploit

Post by MarkCampbell »

Does anyone have a checksum list for all of the files bundled in the nagiosxi 5.8.1 release? I'd like to compare mine to them, find any that might be different.
Top

Return to “Nagios XI”