slopfront
preferences

Solving a runaway memory process (php7-fpm process of 14GB+)

3 points · submitted Apr 2020 by u/DerfK

I have memory_limit set to "128M" confirmed by phpinfo(), and max_execution_time set to 300 seconds. Somehow something is ignoring both of these causing 14GB+ memory usage in a single php-fpm child, by which time the server is no longer responding and has to be killed. Of course, since fpm is killed, it cannot log the offending request since it never finished. All of the other children are normal-sized so I am fairly sure this is caused by a single request and not a lot of large ones or a memory leak over time.

Questions to get to the bottom of this or stop it from bringing everything to a halt:

  1. Is there a way to get the fpm master process to reserve a slot for the php-status page so I can get the list of currently running requests and see what this pid is doing? Failing that, is there ANY other way to get fpm to tell me what the heck is going on right now?
  2. Is there a way to get the fpm master process to enforce a maximum size on a child from outside? I'm guessing something in libxml or another module that ignores memory_limit is going haywire. If it kills the process when it hits say 4GB it would be nice if it could either log the offending request or at least something to let me know when it happened so that I can look for the broken connection in the apache log.
  3. Anyone know any monitoring tools that can watch for oversized processes and react on its own to solve the problem (eg wget localhost/php-status?full then restart)?

4 comments · 100% upvoted · other discussions

comments

wh33t 4 points · Apr 2020

Is this script being called by a cron script or executing with php-cli?

Mika56 1 points · Apr 2020

Is it possible that the script is calling ini_set("memory_limit", $somethingBad)? Your could add ini_set to the list of forbidden functions

AllenJB83 1 points · Apr 2020

Be aware that there's often separate ini files for scripts run from crons / commandline than those run via php-fpm / mod_php. Commandline scripts usually have no max execution time and often higher memory limits.

Note that (as far as I recall) max execution time only applies to a single request, and only to time spent in PHP code - time spent in extensions or waiting for responses from external services is not counted.

If the script is being run as a web request, if php-fpm is dying before completing the request or otherwise not responding you should be able to see this in the web server logs (while there may be a cluster, you should at least be able to narrow down the possibilities).

Does the memory usage grow relatively slowly over time? Or does it suddenly shoot up from almost nothing?

If memory_limit is "being ignored" then the problem is almost certainly with an extension (or rather the library an extension uses) not cleaning up after itself properly.

"One simple trick" to resolve the issue may be to ensure you're using the latest version of PHP and any extensions (especially if you're still on PHP 5).

If the problem is with php-fpm served requests and memory usage growing over time, a "workaround" may be to use the pm.max_requests configuration option to have workers automatically restarted every so often.

DerfK [OP] 1 points · Apr 2020

may be to use the pm.max_requests configuration option to have workers automatically restarted every so often.

Thanks, I've enabled that (set it to 500) and also found request_terminate_timeout and set it to 300s. It appears that this timeout is enforced by fpm itself and might be able to kill a runaway extension if php can't stop itself.

As for cron, I've updated the cli version of the php.ini to have a 2GB memory limit just in case, but since it's a php-fpm process I'm sure it's a request from the internet.

(As for php5, I'm on Debian oldstable, so php7.0. We had other priorities but we're accelerating testing the code on Debian stable php7.3 now)