Bad performance of PHP 8.1 on ARM / Graviton2 AWS instances

[Seldaek]

Frequently asked questions

• I have read Frequently Asked Questions

Describe the bug
We just upgraded to Ubuntu 22.04 and took the chance to switch to ARM / graviton2 AWS instances and we are now seeing much more CPU usage for the same workload (~50% increase in CPU usage). I also noticed much slower HTTP downloads from S3.

I don't know if you can or should do anything here, this is mostly a question (and a way to gather feedback from others perhaps) whether this PPA has ARM-optimized builds or not? Should we rather use something else? Any feedback welcome.

As per AWS PHP should work well if not better on ARM, so I am quite surprised by this finding and I'm hoping it's something we can fix.

Distribution:

• OS: Ubuntu 22.04
• Architecture: aarch64
• Repository: ppa:ondrej/php

Package(s):

php8.1-fpm:
  Installed: 8.1.6-1+ubuntu22.04.1+deb.sury.org+1
  Candidate: 8.1.6-1+ubuntu22.04.1+deb.sury.org+1
  Version table:
 *** 8.1.6-1+ubuntu22.04.1+deb.sury.org+1 500
        500 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy/main arm64 Packages
        100 /var/lib/dpkg/status
     8.1.2-1ubuntu2 500
        500 http://eu-west-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/universe arm64 Packages

[Seldaek]

The S3 issue seems to be handshake related, so it may well be related to the fact Ubuntu 22.04 uses OpenSSL 3.0. Per openssl/openssl#17064 & openssl/openssl#17950 it seems the issue may lie there.

Not sure if this also explains the higher CPU usage we see, as we connect to MySQL over TLS it may well have an overall impact.

[Seldaek]

@oerdnj sorry to bug you here but I spent all day eliminating things so I am now pretty sure this is not due to PHP 8.1 (8.0 is just as CPU intensive), and it's not ARM (I rebuilt one machine on Intel CPU and same..). So it looks like it's Ubuntu 22.04 and something it ships with.. Which makes openssl a likely culprit.

Knowing that, I just wanted to make sure, openssl is not provided by your repo and not bundled as part of your PHP, is that correct? Does a fix depend on Ubuntu's openssl package getting a new release with some amount of perf fixes?

[oerdnj]

Yes, the OpenSSL is not provided from the repository. PHP is compiled with default optimizations provided by the system compiler, I do not do any extra modifications to the build flags here.

[Seldaek]

OK thanks for confirming. I tried pulling OpenSSL 3.0.3 from kinetic packages , but that unfortunately does not seem to help. Edit: It actually does help a bit, but it's not back to openssl 1 level performance.

php -i shows this though so I am a little confused if the upgrade worked:

OpenSSL Library Version => OpenSSL 3.0.3 3 May 2022
OpenSSL Header Version => OpenSSL 3.0.2 15 Mar 2022

[commercebuild-daniel]

We also see very poor performance on ubuntu 22.04 and 24.04 php packages, either stock ubuntu or ppa:ondrej/php

the 20.04 php8.1 builds on ppa:ondrej/php were fine.

Did you find a solution?

[Seldaek]

The best mitigation I've found is to use persistent connections for things like redis, and proxysql or pgbouncer for the database connections so you can do cheap localhost connections without ssl. For http requests I don't have a silver bullet that depends on your use cases, but hopefully ubuntu26.04 will finally address some of this as it should use a newer openssl than the 3.0 series.