Category Archives: System Administration

Turning On MySQL TLS for Sites I Don’t Control

I run a shared hosting platform with a lot of customer WordPress sites on it (thousands), and every one of them talks to a MySQL cluster over the network. I wanted that traffic encrypted and the server’s certificate verified, without editing a single customer site. PHP has no way to do that, so I wrote php-mysqlnd-tls, an extension that turns it on from php.ini:

mysqlnd_tls.enabled = On
mysqlnd_tls.hosts   = "db.example.com"

Only the Client Can Ask

In the MySQL protocol the server speaks first, and the client then decides whether to request TLS before it logs in. The server can accept a plaintext login or refuse it, but it can’t upgrade one.

The mysql command line client asks by default, but PHP doesn’t. On a stock PHP build, both mysqli and PDO connect through mysqlnd (for mysqli it’s been the only driver since PHP 8.2), and mysqlnd uses TLS only when the application requests it: MYSQLI_CLIENT_SSL, mysqli::ssl_set(), or the PDO::MYSQL_ATTR_SSL_* options. There’s no php.ini setting for it.

One Line per Site Is Still Too Many

WordPress can ask, with one line in wp-config.php:

define('MYSQL_CLIENT_FLAGS', MYSQLI_CLIENT_SSL);

That’s one edit per site, in a file the customer owns. Customers restore old backups, migrate sites in from other hosts, and reinstall from scratch, and each of those quietly puts the site back on plaintext. The line also does nothing for sites that aren’t WordPress.

Even where the line is set, WordPress passes the flag and nothing else, and with the flag alone mysqlnd doesn’t verify the server’s certificate. The connection is encrypted, but nothing checks who’s on the other end.

I could have added that line myself. PHP’s auto_prepend_file setting runs a file ahead of every script, so one prepended file could define the constant for every site. That means running my code inside every request to every customer’s site, which is a security issue. It’s a permissions issue too: the file has to be readable by each site’s own user and sit inside each site’s open_basedir, or PHP refuses to open it. And it’s still WordPress only.

The server side is no help. Set require_secure_transport = ON and MySQL refuses every plaintext login, which on day one means every site that hasn’t been edited. Requiring it user by user, with ALTER USER ... REQUIRE SSL, helps with a slow rollout, but it doesn’t solve the other issues: each of those sites still has to ask for TLS.

The Tunnel I Used Before

My old answer was a TLS tunnel that’s invisible to the sites: a proxy on each web server accepts plaintext MySQL connections and carries them inside TLS to the MySQL cluster, where a second proxy unwraps them.

The problem is that every database connection from every site goes through the tunnel, which makes it a single point of failure between two systems that were built to stay up. When it goes down, the web servers and MySQL are both still up, and every site shows a database error anyway.

MySQL also never sees TLS. The tunnel hands mysqld an ordinary plaintext TCP connection, so require_secure_transport and REQUIRE SSL can never be turned on.

Native TLS has neither problem: there’s nothing in between to fail, and both ends know the connection is encrypted, so the server can enforce it. I just needed a way to turn it on.

Moving the Decision to php.ini

mysqlnd has a plugin API that lets an extension replace the methods of its connection object. php-mysqlnd-tls replaces connect with a wrapper: for a host on the list it adds the TLS flag, sets the verification flags and the trust store, and calls the original. That’s about 250 lines of C. It never reads mysqlnd’s internal structures, which is why one source tree builds on every PHP version from 7.4 to 8.6.

For a host on the list, the setting decides what every site gets:

The site What it gets
Asks for nothing TLS, with the certificate and host name verified
Already asks for TLS (the WordPress define) The same; nothing conflicts
Sets its own CA, or asks for no verification The administrator’s trust store and verify setting
Sends a client certificate Kept, so accounts that require one keep working
Connects to a server that doesn’t offer TLS Refused. There’s no fallback to plaintext
Connects to any other host, or over a Unix socket Left exactly as it was

Only the Hosts You List

The policy applies to the hosts named in mysqlnd_tls.hosts and to nothing else. I list my own cluster, so every connection to it is encrypted and verified, whatever the site asked for.

A customer who connects to an external MySQL server is left alone. If their site asks for TLS, it gets TLS with its own settings, and if it connects in plaintext, that still works. It’s their server and their decision. The extension lets me enforce encryption on the servers I control without breaking a site that points somewhere else.

Rolling It Out

Every setting is PHP_INI_SYSTEM, so ini_set(), .htaccess, and .user.ini can’t change it. The administrator can still enable it per site, which makes it easy to roll out a few sites at a time:

<VirtualHost *:80>
    ServerName www.example.com
    php_admin_flag mysqlnd_tls.enabled on
</VirtualHost>

It installs with PIE. Each release carries pre-built binaries for PHP 7.4 through 8.6 on x86_64 and arm64, so most Linux servers don’t need a compiler:

pie install mikepultz/php-mysqlnd-tls

Then Enforce It on the Server

The extension makes PHP ask for TLS, but it can’t stop a client that doesn’t load it: anyone who can run php -n connects however they like, and so does anything that isn’t PHP. The session status shows what a connection actually got:

$row = $mysqli->query("show session status like 'Ssl_cipher'")->fetch_row();

echo ($row[1] === '') ? 'plaintext' : 'encrypted with ' . $row[1];

Once no client is still connecting in plaintext, set require_secure_transport = ON. That’s the setting I couldn’t touch at the start, and it’s on now.

The extension has been running in production for weeks now, and it works great.

Patching PHP’s Two Critical RCEs

On 21 October the PHP project released 8.5.12, 8.4.27, 8.3.36 and 8.2.35 to fix two remote code execution vulnerabilities, CVE-2026-109618 and CVE-2026-109619. Both are rated 9.2 on CVSS v4, and neither needs authentication. Both advisories say updating is the only complete fix.

Fixed Versions

Branch Fixed in
8.5 8.5.12
8.4 8.4.27
8.3 8.3.36
8.2 8.2.35
8.1 and older No upstream release. The PHP security team published minimal source backports for 7.4, 8.0 and 8.1; Debian and Ubuntu users can get fixed builds from packages.sury.org; otherwise check your LTS vendor.

Linux distributions received the patches in advance, so most systems get the fix as a normal package update.

The Two Bugs

CVE-2026-109618 is in out-of-memory handling. PHP aborts on an out-of-memory error with longjmp(). If that happens while it’s still parsing the request body, request data is left half-initialized, and request shutdown later processes it as valid. Crafted multipart requests can trigger this while staying inside PHP’s input limits. The fix removes longjmp() from out-of-memory handling.

CVE-2026-109619 is a use-after-free. A request that’s interrupted by a fatal error, a timeout or running out of memory can leave freed state behind on the worker, and a later request on the same worker can read it before initialization resets it. Confirmed paths include session upload progress, which is on by default and doesn’t need session_start(), and PHP 8.5’s fatal_error_backtraces. The impact includes memory disclosure as well as code execution.

Exposure by Server API

SAPI CVE-2026-109618 CVE-2026-109619
Apache mod_php Code execution validated with default limits, before application code runs Code execution confirmed
PHP-FPM Exploitable by source analysis, not fully demonstrated Code execution validated through session upload progress; depends on the application
php-cgi, FrankenPHP, LiteSpeed Listed as at risk by php.net

Checking What You Run

The CLI and the web server often run different PHP packages, so check the one serving requests:

php -v                      # CLI only
php-fpm -v                  # often versioned: php-fpm8.3 -v
apachectl -M | grep -i php  # is mod_php loaded?
apt policy php8.3-fpm       # Debian/Ubuntu: installed version and its repo

On Debian and Ubuntu, fixed builds reached packages.sury.org before the ppa:ondrej/php Launchpad PPA, so confirm the installed version if you use the PPA. Ondřej Surý is tracking the rollout, including the end-of-life branches, in this issue.

After updating, restart PHP-FPM or Apache so no worker is still running the old build. The full details, including interim settings for systems that can’t update immediately, are in the php.net advisory.

systemd Hardening for Long-Running Daemons

A long-running daemon that talks to the network and a database is the thing an attacker wants most, and it is usually the thing running with the fewest restrictions. systemd ships a large set of sandboxing directives that cost nothing at runtime and are almost never used, mostly because the documentation presents forty options with no indication of which ones matter.

Measure First

systemd will grade the unit for you, which is a better starting point than reading the manual page end to end:

systemd-analyze security tracker.service

You get a per-directive table and an overall exposure score from 0 to 10. A default unit with no hardening scores around 9.6, which the tool cheerfully labels UNSAFE. The score is a rough heuristic rather than a real measure of risk, and it is genuinely useful for ordering the work, since it sorts by exposure and shows what each setting would buy.

The Directives Worth Setting

These are the ones I put on every service, in rough order of value per unit of trouble:

[Service]
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RemoveIPC=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

NoNewPrivileges=yes is the single highest-value line. It blocks setuid escalation for the process and everything it spawns, and almost nothing legitimately needs the alternative.

ProtectSystem=strict mounts the entire filesystem read-only apart from /dev, /proc, and /sys. Combined with ProtectHome=yes, a compromised process cannot write anywhere at all until you grant specific paths back:

ReadWritePaths=/var/lib/tracker /var/log/tracker
RuntimeDirectory=tracker
StateDirectory=tracker

The RuntimeDirectory and StateDirectory forms are better than listing paths by hand, because systemd creates them with the right ownership and cleans up runtime state on stop.

CapabilityBoundingSet= with an empty value drops every capability. Most daemons need none, and the common exception is binding a port below 1024, which needs CAP_NET_BIND_SERVICE added back. Better still, use a socket unit and let systemd hold the privileged socket so the service never needs the capability at all.

RestrictAddressFamilies is worth the thought. A daemon doing DNS and PostgreSQL over TCP needs AF_INET, AF_INET6, and AF_UNIX for the local socket. It has no business creating a raw packet socket or an AF_NETLINK socket, and denying those closes a real avenue.

The Ones That Will Break You

MemoryDenyWriteExecute=yes blocks pages that are both writable and executable. It is excellent against a large class of exploit, and it breaks anything with a JIT. For a C++ daemon it is usually free. For anything embedding a scripting runtime, test carefully.

SystemCallFilter=@system-service is the setting most likely to produce a mysterious failure. The allow list covers what normal services do, and a program using something unusual gets killed with SIGSYS and no useful message. When a service dies immediately after adding it, find the offending call rather than guessing:

journalctl -u tracker.service | grep -i seccomp
auditctl -a always,exit -F arch=b64 -S all -F key=seccomp-debug

ProtectSystem=strict breaks anything writing outside its declared paths, which in practice means log files in unexpected places and PID files in /var/run. Both are better fixed than exempted.

ProtectProc=invisible hides other users’ processes in /proc. Harmless for most daemons and immediately fatal for anything that inspects the process table, including some monitoring agents.

Do It Incrementally

Adding all twenty lines at once and finding the service dead tells you nothing about which line did it. Add the block above minus SystemCallFilter and MemoryDenyWriteExecute, confirm the service still works under real load rather than just starting, then add those two separately.

“Starts successfully” is a weak test. Sandboxing failures show up on code paths that only run occasionally, so the syscall your daemon makes once a day during log rotation is exactly the one the filter blocks. Give a hardened unit a full cycle of whatever periodic work it does before considering it verified.

None of this replaces running as an unprivileged user, keeping the software patched, or putting the database somewhere the internet cannot reach. It is the layer underneath those, and it is roughly twenty lines of configuration that turns a total compromise of the process into a considerably less useful foothold.

PgBouncer, Prepared Statements, and Picking the Right Pool Mode

PgBouncer is the standard answer when an application opens more PostgreSQL connections than the server can usefully service. A process-per-request runtime and a serverless function that dials the database directly get there by completely different routes and land in the same place. The part that goes wrong is the pool mode, because the aggressive setting is the one everybody wants and it silently changes the semantics your code was written against.

Three Modes, One Real Decision

Session pooling assigns a server connection for the life of the client connection. It is safe and it buys you very little, since a worker process holding a connection for the length of a request is the situation you were trying to fix.

Transaction pooling assigns a server connection for the length of a transaction and returns it to the pool at commit. This is the mode that delivers the numbers people install PgBouncer for, and it is the mode with consequences.

Statement pooling returns the connection after every individual statement, which forbids multi-statement transactions entirely. It exists for specific workloads and is almost never what you want.

What Transaction Mode Takes Away

The rule is that anything living outside a transaction stops being reliable, because between two statements you may be on a different backend. That is a longer list than it first appears:

SET / RESET at session scope
LISTEN / NOTIFY
advisory locks taken outside a transaction
WITH HOLD cursors
temporary tables
session-scoped GUCs set by a connection hook

Session-level SET is the one that catches real applications. A framework bootstrap that sets search_path or timezone once on connect works perfectly in development against a direct connection, and in production the setting lands on whichever backend happened to serve that statement. Every later query gets a different backend without it. The failure is intermittent and nearly impossible to reproduce on demand.

Advisory locks are worse, because the failure is silent rather than noisy. A lock taken with pg_advisory_lock() outside a transaction belongs to a session you no longer control. The transaction-scoped variant, pg_advisory_xact_lock(), is released at commit and behaves correctly under transaction pooling. If you use advisory locks for job coordination, that one function name is the difference between working and quietly running the same job twice.

Prepared Statements Are Fixed, with Conditions

For years the answer to prepared statements under transaction pooling was that you could not use them. PgBouncer 1.21, released in October 2023, changed that. Set max_prepared_statements to a non-zero value and PgBouncer tracks named prepared statements itself, rewriting them to internal names and re-preparing them on whichever backend a client lands on:

[pgbouncer]
pool_mode = transaction
max_prepared_statements = 200

The value is the size of an LRU cache of statements kept on each server connection, so it wants to be at least as large as the number of distinct statements a typical request issues. Zero disables the feature and restores the old behavior.

The condition attached is important. This only works for protocol-level prepared statements, the ones sent through the extended query protocol. A literal PREPARE foo AS SELECT ... sent as a plain text query is invisible to PgBouncer and still breaks, because from the outside it is just another statement.

For PHP specifically, this means checking what PDO is actually doing. With ATTR_EMULATE_PREPARES left on, PDO interpolates parameters client-side and sends plain SQL, so there are no real prepared statements and nothing to break. Turn emulation off and you get genuine protocol-level prepares, which is what you want for both correctness and plan reuse, and which is exactly the case max_prepared_statements exists to handle.

$pdo = new PDO($dsn, $user, $pass, [
    PDO::ATTR_EMULATE_PREPARES => false,
    PDO::ATTR_ERRMODE          => PDO::ERRMODE_EXCEPTION,
]);

How I Choose

Start with transaction mode, then go looking for the session state your application depends on rather than waiting for it to surface. Grep for LISTEN, for pg_advisory_lock, for temporary tables, and for anything setting a GUC on connect. Move search_path out of a connection hook and into the connection string, where PgBouncer passes it through as part of the startup parameters and it survives correctly.

Keep a second pool in session mode on a different port for the small number of things that genuinely need a stable session, which is usually a migration runner and whatever handles LISTEN. Two pools with clear rules beats one pool with exceptions nobody remembers.

And size the pool against what PostgreSQL can actually do, since the point of pooling is to keep the database out of the region where it spends more time context switching than working. A pool larger than the database can service just moves the queue.

MTA-STS and TLS-RPT: Enforcing TLS on Inbound Mail

SMTP encryption is opportunistic by default, which means it is optional, which means it is strippable. A sending server connects, looks for STARTTLS in the EHLO response, and upgrades if it sees it. Remove that one line in transit and the sender falls back to plaintext without complaint, because it has no way to know TLS was ever supposed to happen. MTA-STS is how you tell it.

Why STARTTLS Alone Is Not Enough

The gap is that opportunistic TLS has no expectation to violate. A sender that fails to negotiate TLS has no basis to refuse delivery, since plenty of legitimate mail servers still do not offer it. Certificate validation is usually skipped for the same reason: a mail server presenting a self-signed or expired certificate is common enough that treating it as fatal would break real delivery.

So the default posture is an encrypted channel to an unverified party, downgradeable by anyone in the path. MTA-STS, specified in RFC 8461, lets a receiving domain publish a policy saying TLS is required, the certificate must validate, and here are the hostnames allowed to receive mail. Senders that implement it will refuse to deliver rather than fall back.

The Three Pieces

First, a DNS TXT record at the _mta-sts label telling senders a policy exists:

_mta-sts.example.com.  IN  TXT  "v=STSv1; id=20260916120000Z;"

The id is the whole mechanism for cache invalidation. Senders compare it against the one they saw last time, and they only re-fetch the policy when it changes. Bump it on every policy edit or your change will not be noticed until caches expire on their own.

Second, the policy itself, served over HTTPS from a specific host and path:

https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 604800

mode takes enforce, testing, or none. Every MX that can receive mail for the domain needs a line, and a wildcard like *.example.com matches only the leftmost label, so it covers mail.example.com and not foo.bar.example.com. max_age is in seconds with a ceiling of 31557600, about a year.

Third, and this is the part people trip over: the mta-sts host serving that file needs a valid, publicly trusted certificate of its own. The policy is only as trustworthy as the HTTPS connection that delivered it, so a sender that cannot validate that certificate discards the policy entirely. You have now made your mail delivery depend on a certificate on a host that has nothing else to do with mail, which is exactly the kind of endpoint that quietly expires.

Start in Testing Mode

Publishing mode: enforce as the first step is how you find out about your forgotten backup MX by having mail to it rejected. mode: testing makes senders evaluate the policy, report failures, and deliver anyway.

Leave it there long enough to see a full cycle of your real traffic. A week is reasonable, longer if you have partners who send in bursts. The reports are the point of the exercise, which brings up the other half.

TLS-RPT Tells You What Broke

RFC 8460 defines a companion record that asks senders to report what happened. It is cheap to add and it is the only feedback channel you get:

_smtp._tls.example.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:tlsrpt@example.com"

Reports arrive as JSON, daily, from each sending organization that implements it. The useful part is the failure detail:

{
  "organization-name": "Example Sender Inc",
  "date-range": { "start-datetime": "2026-09-16T00:00:00Z",
                  "end-datetime":   "2026-09-16T23:59:59Z" },
  "report-id": "2026-09-16T00:00:00Z_example.com",
  "policies": [{
    "policy": { "policy-type": "sts", "policy-domain": "example.com" },
    "summary": { "total-successful-session-count": 8214,
                 "total-failure-session-count": 17 },
    "failure-details": [{
      "result-type": "certificate-host-mismatch",
      "sending-mta-ip": "203.0.113.9",
      "receiving-mx-hostname": "mail2.example.com",
      "failed-session-count": 17
    }]
  }]
}

The result-type values are specific enough to act on directly. starttls-not-supported, certificate-expired, certificate-host-mismatch, certificate-not-trusted, validation-failure, and on the policy side sts-policy-fetch-error, sts-policy-invalid, and sts-webpki-invalid. That last group points at your policy host rather than your mail servers, which is a distinction worth internalizing before you start debugging the wrong machine.

The Ways This Bites

Caching cuts both ways. Once a sender has your policy with a week-long max_age, a mistake persists for that long even after you fix the file, unless the sender happens to re-check the id. Adding an MX means updating the policy and bumping the id before the new host starts receiving, and doing it in that order.

The certificate on the mta-sts web host is the failure nobody plans for. It is usually a static file on a server that gets less attention than the mail infrastructure, and when it expires, compliant senders stop trusting your policy. Whether that means mail is deferred or the policy is simply ignored depends on the sender.

Your MX certificates now genuinely have to be correct as well. Names have to match what the policy lists, chains have to be complete, and expiry stops being cosmetic. That is the point of turning it on, and it is worth knowing before you flip to enforce.

To check what you have published, the MTA-STS checker on mrdns.com fetches the record and the policy together and tells you whether they agree. For the mail server configuration underneath it, goodtls.com has per-application guides covering Postfix, Exim, Dovecot, and the rest.