[Bug]: Server hangs as php-fpm reach pm.max_children limit

[ThibautPlg]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Hello,
I'm administrating multiple Nextcloud 25 instances, and I'm slowly upgrading to Nextcloud 26. However, after some days (some hours sometimes), each and every instance upgraded to Nextcloud 26 crash due to the php-fpm pm.max_children limit being reached.
I then need to restart php-fpm and everything goes normal until the next crash.

Additional context:

• The servers are only hosting one Nextcloud server, with php 8.1
• php-fpm max_children and other configuration values have been customized to match the available RAM of the host (between 32 to 64 childs allowed)
• The php-fpm max children have never been a problem as far as I recall (some instances are older than NC20). The problem also occurs on newly installed testing instances.
• The ram consumption is average, we're not maxed
• I've tried with php8.2, same results

systemctl status php-fpm.service output when server is down:

● php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-06-28 14:30:57 CEST; 24h ago
  Process: 3345857 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 462753 (php-fpm)
   Status: "Processes active: 64, idle: 2, Requests: 127550, slow: 0, Traffic: 0req/sec"
    Tasks: 67 (limit: 47993)
   Memory: 1.7G
   CGroup: /system.slice/php-fpm.service
           ├─ 462753 php-fpm: master process (/etc/php-fpm.conf)
           ├─ 613484 php-fpm: pool nextcloud
           ├─ 613485 php-fpm: pool nextcloud
           ├─ 613486 php-fpm: pool nextcloud
           ├─ 613487 php-fpm: pool nextcloud
           ├─ 613488 php-fpm: pool nextcloud
           ├─ 613489 php-fpm: pool nextcloud
(the list goes on)

I feel like some processes are idle (hanged?) and never stopped.

Am I the only one facing this issue? Why does it occurs only with NC26? What has changed regarding php processes?

Best regards,

Steps to reproduce

1. Upgrade to NC26 or install a fresh server
2. Wait a little bit
3. Observe php-fpm being overwhelmed

Expected behavior

Same as before, Nextcloud (php-fpm?) should remove children.

Installation method

None

Nextcloud Server version

26

Operating system

RHEL/CentOS

PHP engine version

PHP 8.1

Web server

Nginx

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

[joshtrichards]

Hi @ThibautPlg:

There are a lot of possibilities, but you didn't provide a complete Issue form. :-)

The most important items to check off the top of my head that might provide clues:

• your php-fpm logs
• your nextcloud.log
• your php-fpm status page (particularly in full mode)
• your php-fpm pool configuration
• your nginx configuration (also worth comparing against the one in the NC manual since it's periodically updated in-between NC versions)
• which NC apps are active

Are all of these instances essentially similarly configured/built? While anything is possible, chances are this is some sort of local environment interaction.

I would suggest posting this in the Nextcloud Help Forum first. NC26 has been out awhile and I haven't seen rampant reports of new php-fpm related issues personally.

[bjo81]

@ThibautPlg Does this issue disappear when you disable previews? We have an instance where php-fpm is stuck with requests like GET /core/preview?fileId=1151626&c=25fa3ba9d97519e2dd8f7ef2595bdf02&x=500&y=500&forceIcon=0&a=1 HTTP/2.0. Even running the preview generator app get's stuck on generating a preview of a PDF like "Nextcloud Flyer.pdf" then, so it seems the whole preview generation is stuck. All php-fpm processes hang at semop(1, [{0, -1, SEM_UNDO}],
The issue first appeared with 26.0.0, so it could be related to the new preview generation code. With < 26.0.0 15 php-fpm workers were fine, but now eben 270 are not enough as they alwas hang and never get killed.

[ThibautPlg]

Hi
Sorry for my absence of answers to @joshtrichards , I had a lot on my plate lately and this subject kind of went in background.

This is an example of a regular work day for my users, as you can see the fpm processes suddenly spikes until I manually reload them and everything goes right until the next time the server has a mood change.
I haven't noticed anything linked to the previews, requests are quite random and the slow.log contains entries for all kinds of scripts and nothing rings a bell for me.

In the end we "fixed" the problem by adding the following line to our php-fpm config : request_terminate_timeout = 5m.

[marc4s]

on my instance this started with 25.0.9 or 25.0.10. ltoday I upgraded to 26.0.5, I will check in the next days the issue persists...

preview was all the disabled

[diego-treitos]

I am experiencing this issue too and I also noticed it since the upgrade to 25.x.x (not sure what version). I administer several NC servers and all of them have the same behavior. While request_terminate_timeout = 5m is a working workaround, I think it is only a patch and I guess it might have an impact on performance? Anyway this looks like a bug on NC as it started happening after an upgrade.

[diego-treitos]

Actually setting request_terminate_timeout = 5m creates a problem when syncing big files. If you check the documentation for uploading big files (https://docs.nextcloud.com/server/20/admin_manual/configuration_files/big_file_upload_configuration.html) you see that they recommend to raise timeouts even to 1 hour. This means that if your requests terminate after 5 minutes, your big files won't sync.

I think the problem might be related to nextcloud server either not closing database connections or not recycling them in future requests because I've observed that database connections increase at the same pace that workers.

This issue is causing a lot of troubles in all my instances. I am surprised that it is not getting more attention.

[ThibautPlg]

The request_terminate_timeout is indeed only a workaround. We've also noticed a high increase in mariadb connections prior to a total overflow of php-fpm processes.
Quite hard to debug. Thanks for your comment though, glad (kind of) to not be the only one affected by this behavior.

[robert-scheck]

I see exactly the same result like @ThibautPlg reported, however on a slightly different system:

• CentOS 7 (fully up-to-date)
• PHP 8.1 from Remi's Safe RPM repository
• Nextcloud 27.1.2

However, it is mod_php instead of PHP-FPM, but all Apache webserver processes are stuck at semop() as well, when this issue occurs. Using strace, I unfortunately can gather this:

semop(12, [{0, -1, SEM_UNDO}], 1

And on this system, the pm.max_children limit isn't hit (because no PHP-FPM), but instead the maximum of Apache webserver processes (httpd) or the maximum of MariaDB connections (depending on where you set a higher limit) at mysqld.