Hello, since we updated to the latest version, we are having a RAM use problem, its use is slowly increasing until it ends and it begins to give ram usage alerts, we restart the server and it starts again, this takes days but in the graph can be seen as it increases as time passes. What could be happening?
I am talking about my nagiosxi server, in the old version it had half the resources and it worked without problems
constant increase in memory usage
-
supportcl
- Posts: 39
- Joined: Wed Oct 07, 2020 12:19 pm
constant increase in memory usage
You do not have the required permissions to view the files attached to this post.
-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: constant increase in memory usage
What specifically is the process taking up all the RAM?
If you don't know how to figure that out, you can PM me a profile so I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.
If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:
Then send me the resulting /usr/local/nagiosxi/var/components/profile.zip file.
If the profile script fails, please include the ENTIRE output.
If you don't know how to figure that out, you can PM me a profile so I can diagnose further. Get one by going to Admin (top menu) => System Profile (in the left menu), then clicking the blue button.
If you're unable to generate the the profile through the web interface, please try generating it from the command line by running these commands as root:
Code: Select all
rm -rf /usr/local/nagiosxi/var/components/profile*
/usr/local/nagiosxi/scripts/components/getprofile.sh SUPPORTIf the profile script fails, please include the ENTIRE output.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
supportcl
- Posts: 39
- Joined: Wed Oct 07, 2020 12:19 pm
Re: constant increase in memory usage
I have sent you the requested profile file.
The strange thing is that I could not see what is consuming the RAM, now I had to restart the system since we were having problems.
Check the server log file
The strange thing is that I could not see what is consuming the RAM, now I had to restart the system since we were having problems.
Check the server log file
You do not have the required permissions to view the files attached to this post.
-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: constant increase in memory usage
Looks like mysqld was using 4.9GB of memory or more when the OOM killer shot it in the head, and even after a reboot it's still using 118% of a CPU. Maybe there's a rogue index somewhere.
What's the output from the following command?
What's the output from the following command?
Code: Select all
mysql -unagiosxi -pn@gweb --table <<< "select table_name as 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' from information_schema.TABLES where table_schema in ('nagios', 'nagiosql', 'nagiosxi');"If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
supportcl
- Posts: 39
- Joined: Wed Oct 07, 2020 12:19 pm
Re: constant increase in memory usage
Code: Select all
[root@nagiosxi log]# mysql -unagiosxi -pn@gweb --table <<< "select table_name as 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' from information_schema.TABLES where table_schema in ('nagios', 'nagiosql', 'nagiosxi');"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------------------------+------------+
| Table | Size in MB |
+-----------------------------+------------+
| xi_auditlog | 7.03 |
| xi_auth_tokens | 1.56 |
| xi_cmp_ccm_backups | 0.02 |
| xi_cmp_favorites | 0.03 |
| xi_cmp_nagiosbpi_backups | 0.27 |
| xi_cmp_scheduledreports_log | 0.02 |
| xi_cmp_trapdata | 0.03 |
| xi_cmp_trapdata_log | 0.03 |
| xi_commands | 0.01 |
| xi_deploy_agents | 0.02 |
| xi_deploy_jobs | 0.02 |
| xi_eventqueue | 0.10 |
| xi_events | 0.14 |
| xi_incidents | 0.00 |
| xi_meta | 8.31 |
| xi_mibs | 0.05 |
| xi_options | 0.03 |
| xi_sessions | 0.03 |
| xi_sysstat | 0.01 |
| xi_usermeta | 0.56 |
| xi_users | 0.02 |
+-----------------------------+------------+
[root@nagiosxi log]#-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: constant increase in memory usage
Try running the database repair script, and let me know if that is successful. Run the following as root from the terminal.
See here for complete instructions: run the database repair
Code: Select all
/usr/local/nagiosxi/scripts/repair_databases.shIf you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: constant increase in memory usage
Did you get a chance to do this? Let us know if you have any related questions or if we're okay to lock this up and mark it as resolved.
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
jweijters
- Posts: 63
- Joined: Thu Feb 06, 2020 3:50 am
Re: constant increase in memory usage
Hi I have similar problem, however memory usage goes pretty fast.
It is the php-fpm processes using up the memory:
I ran a repair database. for a moment the usage was gone, however after approx 15 minutes the memory usage is very high again.
see the 4 processes using 15.5% memory
It is the php-fpm processes using up the memory:
I ran a repair database. for a moment the usage was gone, however after approx 15 minutes the memory usage is very high again.
see the 4 processes using 15.5% memory
Code: Select all
$ free -h
total used free shared buff/cache available
Mem: 15G 12G 1.4G 262M 1.8G 2.6G
Swap: 5.9G 647M 5.2G
$ ps aux |grep -i php-fpm
apache 625 0.5 0.2 547104 36820 ? S 07:12 0:10 php-fpm: pool www
apache 633 2.7 15.4 3023704 2513976 ? S 07:12 0:48 php-fpm: pool www
apache 634 0.6 0.2 545736 36040 ? R 07:12 0:12 php-fpm: pool www
apache 645 0.5 0.2 544676 35080 ? S 07:12 0:10 php-fpm: pool www
apache 1433 2.5 15.5 3028084 2516676 ? S 07:13 0:43 php-fpm: pool www
apache 1770 0.5 0.2 543236 33880 ? S 07:14 0:08 php-fpm: pool www
apache 1793 0.5 0.2 544872 35892 ? S 07:14 0:09 php-fpm: pool www
apache 1798 0.6 0.2 546440 37124 ? S 07:14 0:11 php-fpm: pool www
apache 1799 0.6 0.2 546080 34860 ? S 07:14 0:10 php-fpm: pool www
apache 1822 0.5 0.4 577280 67488 ? S 07:14 0:09 php-fpm: pool www
apache 1823 0.5 0.2 544316 32740 ? S 07:14 0:09 php-fpm: pool www
apache 1824 0.5 0.2 546120 35404 ? S 07:14 0:09 php-fpm: pool www
apache 1825 0.5 0.2 631816 39812 ? S 07:14 0:09 php-fpm: pool www
apache 2107 0.5 0.2 544692 35060 ? S 07:14 0:10 php-fpm: pool www
apache 9366 2.8 15.5 3031320 2519108 ? S 07:18 0:40 php-fpm: pool www
apache 15859 3.8 15.4 3025812 2515480 ? S 07:24 0:40 php-fpm: pool www
root 33761 0.0 0.0 112816 984 pts/1 R+ 07:42 0:00 grep --color=auto -i php-fpm
root 201405 0.0 0.0 525328 3068 ? Ss 06:25 0:00 php-fpm: master process (/etc/php-fpm.conf)
apache 245255 0.4 0.2 543424 33824 ? S 07:10 0:09 php-fpm: pool www
apache 245258 0.4 0.2 544884 35316 ? S 07:11 0:08 php-fpm: pool www
apache 245259 0.5 0.2 550256 40580 ? S 07:11 0:09 php-fpm: pool www
apache 245260 0.5 0.2 543576 33856 ? S 07:11 0:11 php-fpm: pool www
-
dchurch
- Posts: 858
- Joined: Wed Oct 07, 2020 12:46 pm
- Location: Yo mama
Re: constant increase in memory usage
PHP is ballooning out to ~500MB-3GB per process, and your PHP-FPM configuration is either set to spawn too many of them, or they're not being reaped. My idea is to change how the children are created, and reap them more effectively.
Edit under the [www] heading in /etc/php-fpm.d/www.conf (replace and/or uncomment values rather than adding them):
Edit under the [www] heading in /etc/php-fpm.d/www.conf (replace and/or uncomment values rather than adding them):
Code: Select all
[www]
# Now directives under this apply to the "www" pool, same as your screenshot
# ...
pm = ondemand
pm.max_children = 5
pm.process_idle_timeout = 10s
pm.max_requests = 500
If you didn't get an 8% raise over the course of the pandemic, you took a pay cut.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
Discussion of wages is protected speech under the National Labor Relations Act, and no employer can tell you you can't disclose your pay with your fellow employees.
-
jweijters
- Posts: 63
- Joined: Thu Feb 06, 2020 3:50 am
Re: constant increase in memory usage
I Adjusted these Parameters,
However it doesn't help.
Because it is the php-fpm, which is started from the http-server so browser initiated, how can I find out which user is causing this high amount of memory usage?
kind regards,
Joris Weijters
However it doesn't help.
Because it is the php-fpm, which is started from the http-server so browser initiated, how can I find out which user is causing this high amount of memory usage?
kind regards,
Joris Weijters