Category Archives: System Administration

Patching PHP’s Two Critical RCEs

On 21 October the PHP project released 8.5.12, 8.4.27, 8.3.36 and 8.2.35 to fix two remote code execution vulnerabilities, CVE-2026-109618 and CVE-2026-109619. Both are rated 9.2 on CVSS v4, and neither needs authentication. Both advisories say updating is the only complete fix.

Fixed Versions

Branch Fixed in
8.5 8.5.12
8.4 8.4.27
8.3 8.3.36
8.2 8.2.35
8.1 and older No upstream release. The PHP security team published minimal source backports for 7.4, 8.0 and 8.1; Debian and Ubuntu users can get fixed builds from packages.sury.org; otherwise check your LTS vendor.

Linux distributions received the patches in advance, so most systems get the fix as a normal package update.

The Two Bugs

CVE-2026-109618 is in out-of-memory handling. PHP aborts on an out-of-memory error with longjmp(). If that happens while it’s still parsing the request body, request data is left half-initialized, and request shutdown later processes it as valid. Crafted multipart requests can trigger this while staying inside PHP’s input limits. The fix removes longjmp() from out-of-memory handling.

CVE-2026-109619 is a use-after-free. A request that’s interrupted by a fatal error, a timeout or running out of memory can leave freed state behind on the worker, and a later request on the same worker can read it before initialization resets it. Confirmed paths include session upload progress, which is on by default and doesn’t need session_start(), and PHP 8.5’s fatal_error_backtraces. The impact includes memory disclosure as well as code execution.

Exposure by Server API

SAPI CVE-2026-109618 CVE-2026-109619
Apache mod_php Code execution validated with default limits, before application code runs Code execution confirmed
PHP-FPM Exploitable by source analysis, not fully demonstrated Code execution validated through session upload progress; depends on the application
php-cgi, FrankenPHP, LiteSpeed Listed as at risk by php.net

Checking What You Run

The CLI and the web server often run different PHP packages, so check the one serving requests:

php -v                      # CLI only
php-fpm -v                  # often versioned: php-fpm8.3 -v
apachectl -M | grep -i php  # is mod_php loaded?
apt policy php8.3-fpm       # Debian/Ubuntu: installed version and its repo

On Debian and Ubuntu, fixed builds reached packages.sury.org before the ppa:ondrej/php Launchpad PPA, so confirm the installed version if you use the PPA. Ondřej Surý is tracking the rollout, including the end-of-life branches, in this issue.

After updating, restart PHP-FPM or Apache so no worker is still running the old build. The full details, including interim settings for systems that can’t update immediately, are in the php.net advisory.

systemd Hardening for Long-Running Daemons

A long-running daemon that talks to the network and a database is the thing an attacker wants most, and it is usually the thing running with the fewest restrictions. systemd ships a large set of sandboxing directives that cost nothing at runtime and are almost never used, mostly because the documentation presents forty options with no indication of which ones matter.

Measure First

systemd will grade the unit for you, which is a better starting point than reading the manual page end to end:

systemd-analyze security tracker.service

You get a per-directive table and an overall exposure score from 0 to 10. A default unit with no hardening scores around 9.6, which the tool cheerfully labels UNSAFE. The score is a rough heuristic rather than a real measure of risk, and it is genuinely useful for ordering the work, since it sorts by exposure and shows what each setting would buy.

The Directives Worth Setting

These are the ones I put on every service, in rough order of value per unit of trouble:

[Service]
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RemoveIPC=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

NoNewPrivileges=yes is the single highest-value line. It blocks setuid escalation for the process and everything it spawns, and almost nothing legitimately needs the alternative.

ProtectSystem=strict mounts the entire filesystem read-only apart from /dev, /proc, and /sys. Combined with ProtectHome=yes, a compromised process cannot write anywhere at all until you grant specific paths back:

ReadWritePaths=/var/lib/tracker /var/log/tracker
RuntimeDirectory=tracker
StateDirectory=tracker

The RuntimeDirectory and StateDirectory forms are better than listing paths by hand, because systemd creates them with the right ownership and cleans up runtime state on stop.

CapabilityBoundingSet= with an empty value drops every capability. Most daemons need none, and the common exception is binding a port below 1024, which needs CAP_NET_BIND_SERVICE added back. Better still, use a socket unit and let systemd hold the privileged socket so the service never needs the capability at all.

RestrictAddressFamilies is worth the thought. A daemon doing DNS and PostgreSQL over TCP needs AF_INET, AF_INET6, and AF_UNIX for the local socket. It has no business creating a raw packet socket or an AF_NETLINK socket, and denying those closes a real avenue.

The Ones That Will Break You

MemoryDenyWriteExecute=yes blocks pages that are both writable and executable. It is excellent against a large class of exploit, and it breaks anything with a JIT. For a C++ daemon it is usually free. For anything embedding a scripting runtime, test carefully.

SystemCallFilter=@system-service is the setting most likely to produce a mysterious failure. The allow list covers what normal services do, and a program using something unusual gets killed with SIGSYS and no useful message. When a service dies immediately after adding it, find the offending call rather than guessing:

journalctl -u tracker.service | grep -i seccomp
auditctl -a always,exit -F arch=b64 -S all -F key=seccomp-debug

ProtectSystem=strict breaks anything writing outside its declared paths, which in practice means log files in unexpected places and PID files in /var/run. Both are better fixed than exempted.

ProtectProc=invisible hides other users’ processes in /proc. Harmless for most daemons and immediately fatal for anything that inspects the process table, including some monitoring agents.

Do It Incrementally

Adding all twenty lines at once and finding the service dead tells you nothing about which line did it. Add the block above minus SystemCallFilter and MemoryDenyWriteExecute, confirm the service still works under real load rather than just starting, then add those two separately.

“Starts successfully” is a weak test. Sandboxing failures show up on code paths that only run occasionally, so the syscall your daemon makes once a day during log rotation is exactly the one the filter blocks. Give a hardened unit a full cycle of whatever periodic work it does before considering it verified.

None of this replaces running as an unprivileged user, keeping the software patched, or putting the database somewhere the internet cannot reach. It is the layer underneath those, and it is roughly twenty lines of configuration that turns a total compromise of the process into a considerably less useful foothold.

PgBouncer, Prepared Statements, and Picking the Right Pool Mode

PgBouncer is the standard answer when an application opens more PostgreSQL connections than the server can usefully service. A process-per-request runtime and a serverless function that dials the database directly get there by completely different routes and land in the same place. The part that goes wrong is the pool mode, because the aggressive setting is the one everybody wants and it silently changes the semantics your code was written against.

Three Modes, One Real Decision

Session pooling assigns a server connection for the life of the client connection. It is safe and it buys you very little, since a worker process holding a connection for the length of a request is the situation you were trying to fix.

Transaction pooling assigns a server connection for the length of a transaction and returns it to the pool at commit. This is the mode that delivers the numbers people install PgBouncer for, and it is the mode with consequences.

Statement pooling returns the connection after every individual statement, which forbids multi-statement transactions entirely. It exists for specific workloads and is almost never what you want.

What Transaction Mode Takes Away

The rule is that anything living outside a transaction stops being reliable, because between two statements you may be on a different backend. That is a longer list than it first appears:

SET / RESET at session scope
LISTEN / NOTIFY
advisory locks taken outside a transaction
WITH HOLD cursors
temporary tables
session-scoped GUCs set by a connection hook

Session-level SET is the one that catches real applications. A framework bootstrap that sets search_path or timezone once on connect works perfectly in development against a direct connection, and in production the setting lands on whichever backend happened to serve that statement. Every later query gets a different backend without it. The failure is intermittent and nearly impossible to reproduce on demand.

Advisory locks are worse, because the failure is silent rather than noisy. A lock taken with pg_advisory_lock() outside a transaction belongs to a session you no longer control. The transaction-scoped variant, pg_advisory_xact_lock(), is released at commit and behaves correctly under transaction pooling. If you use advisory locks for job coordination, that one function name is the difference between working and quietly running the same job twice.

Prepared Statements Are Fixed, with Conditions

For years the answer to prepared statements under transaction pooling was that you could not use them. PgBouncer 1.21, released in October 2023, changed that. Set max_prepared_statements to a non-zero value and PgBouncer tracks named prepared statements itself, rewriting them to internal names and re-preparing them on whichever backend a client lands on:

[pgbouncer]
pool_mode = transaction
max_prepared_statements = 200

The value is the size of an LRU cache of statements kept on each server connection, so it wants to be at least as large as the number of distinct statements a typical request issues. Zero disables the feature and restores the old behavior.

The condition attached is important. This only works for protocol-level prepared statements, the ones sent through the extended query protocol. A literal PREPARE foo AS SELECT ... sent as a plain text query is invisible to PgBouncer and still breaks, because from the outside it is just another statement.

For PHP specifically, this means checking what PDO is actually doing. With ATTR_EMULATE_PREPARES left on, PDO interpolates parameters client-side and sends plain SQL, so there are no real prepared statements and nothing to break. Turn emulation off and you get genuine protocol-level prepares, which is what you want for both correctness and plan reuse, and which is exactly the case max_prepared_statements exists to handle.

$pdo = new PDO($dsn, $user, $pass, [
    PDO::ATTR_EMULATE_PREPARES => false,
    PDO::ATTR_ERRMODE          => PDO::ERRMODE_EXCEPTION,
]);

How I Choose

Start with transaction mode, then go looking for the session state your application depends on rather than waiting for it to surface. Grep for LISTEN, for pg_advisory_lock, for temporary tables, and for anything setting a GUC on connect. Move search_path out of a connection hook and into the connection string, where PgBouncer passes it through as part of the startup parameters and it survives correctly.

Keep a second pool in session mode on a different port for the small number of things that genuinely need a stable session, which is usually a migration runner and whatever handles LISTEN. Two pools with clear rules beats one pool with exceptions nobody remembers.

And size the pool against what PostgreSQL can actually do, since the point of pooling is to keep the database out of the region where it spends more time context switching than working. A pool larger than the database can service just moves the queue.

MTA-STS and TLS-RPT: Enforcing TLS on Inbound Mail

SMTP encryption is opportunistic by default, which means it is optional, which means it is strippable. A sending server connects, looks for STARTTLS in the EHLO response, and upgrades if it sees it. Remove that one line in transit and the sender falls back to plaintext without complaint, because it has no way to know TLS was ever supposed to happen. MTA-STS is how you tell it.

Why STARTTLS Alone Is Not Enough

The gap is that opportunistic TLS has no expectation to violate. A sender that fails to negotiate TLS has no basis to refuse delivery, since plenty of legitimate mail servers still do not offer it. Certificate validation is usually skipped for the same reason: a mail server presenting a self-signed or expired certificate is common enough that treating it as fatal would break real delivery.

So the default posture is an encrypted channel to an unverified party, downgradeable by anyone in the path. MTA-STS, specified in RFC 8461, lets a receiving domain publish a policy saying TLS is required, the certificate must validate, and here are the hostnames allowed to receive mail. Senders that implement it will refuse to deliver rather than fall back.

The Three Pieces

First, a DNS TXT record at the _mta-sts label telling senders a policy exists:

_mta-sts.example.com.  IN  TXT  "v=STSv1; id=20260916120000Z;"

The id is the whole mechanism for cache invalidation. Senders compare it against the one they saw last time, and they only re-fetch the policy when it changes. Bump it on every policy edit or your change will not be noticed until caches expire on their own.

Second, the policy itself, served over HTTPS from a specific host and path:

https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 604800

mode takes enforce, testing, or none. Every MX that can receive mail for the domain needs a line, and a wildcard like *.example.com matches only the leftmost label, so it covers mail.example.com and not foo.bar.example.com. max_age is in seconds with a ceiling of 31557600, about a year.

Third, and this is the part people trip over: the mta-sts host serving that file needs a valid, publicly trusted certificate of its own. The policy is only as trustworthy as the HTTPS connection that delivered it, so a sender that cannot validate that certificate discards the policy entirely. You have now made your mail delivery depend on a certificate on a host that has nothing else to do with mail, which is exactly the kind of endpoint that quietly expires.

Start in Testing Mode

Publishing mode: enforce as the first step is how you find out about your forgotten backup MX by having mail to it rejected. mode: testing makes senders evaluate the policy, report failures, and deliver anyway.

Leave it there long enough to see a full cycle of your real traffic. A week is reasonable, longer if you have partners who send in bursts. The reports are the point of the exercise, which brings up the other half.

TLS-RPT Tells You What Broke

RFC 8460 defines a companion record that asks senders to report what happened. It is cheap to add and it is the only feedback channel you get:

_smtp._tls.example.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:tlsrpt@example.com"

Reports arrive as JSON, daily, from each sending organization that implements it. The useful part is the failure detail:

{
  "organization-name": "Example Sender Inc",
  "date-range": { "start-datetime": "2026-09-16T00:00:00Z",
                  "end-datetime":   "2026-09-16T23:59:59Z" },
  "report-id": "2026-09-16T00:00:00Z_example.com",
  "policies": [{
    "policy": { "policy-type": "sts", "policy-domain": "example.com" },
    "summary": { "total-successful-session-count": 8214,
                 "total-failure-session-count": 17 },
    "failure-details": [{
      "result-type": "certificate-host-mismatch",
      "sending-mta-ip": "203.0.113.9",
      "receiving-mx-hostname": "mail2.example.com",
      "failed-session-count": 17
    }]
  }]
}

The result-type values are specific enough to act on directly. starttls-not-supported, certificate-expired, certificate-host-mismatch, certificate-not-trusted, validation-failure, and on the policy side sts-policy-fetch-error, sts-policy-invalid, and sts-webpki-invalid. That last group points at your policy host rather than your mail servers, which is a distinction worth internalizing before you start debugging the wrong machine.

The Ways This Bites

Caching cuts both ways. Once a sender has your policy with a week-long max_age, a mistake persists for that long even after you fix the file, unless the sender happens to re-check the id. Adding an MX means updating the policy and bumping the id before the new host starts receiving, and doing it in that order.

The certificate on the mta-sts web host is the failure nobody plans for. It is usually a static file on a server that gets less attention than the mail infrastructure, and when it expires, compliant senders stop trusting your policy. Whether that means mail is deferred or the policy is simply ignored depends on the sender.

Your MX certificates now genuinely have to be correct as well. Names have to match what the policy lists, chains have to be complete, and expiry stops being cosmetic. That is the point of turning it on, and it is worth knowing before you flip to enforce.

To check what you have published, the MTA-STS checker on mrdns.com fetches the record and the policy together and tells you whether they agree. For the mail server configuration underneath it, goodtls.com has per-application guides covering Postfix, Exim, Dovecot, and the rest.

Partitioning a Table That Never Stops Growing

Most systems I have worked on end up with a handful of tables that only ever grow. Event history, audit trails, request logs, anything append-only. None of them are interesting individually, and together they eventually dominate the maintenance budget: vacuum takes longer every month, index rebuilds turn into scheduled events, and a query that only wants last week has to reason about years of rows to prove it.

Declarative partitioning is the fix, and PostgreSQL has been good at it since 12. The parts worth writing down are how to get an existing table into it without downtime, and how to stop hand-managing it afterwards.

Partition on the Column You Actually Filter

Range partitioning by time is the obvious choice for append-only data, and the partition key has to appear in your WHERE clauses or you gain nothing. Partition by occurred_at while every query filters on account_id and each one still touches every partition.

CREATE TABLE events (
    id bigint GENERATED ALWAYS AS IDENTITY,
    account_id integer NOT NULL,
    occurred_at timestamptz NOT NULL,
    event_type smallint NOT NULL,
    PRIMARY KEY (id, occurred_at)
) PARTITION BY RANGE (occurred_at);

Note the primary key. Every unique constraint on a partitioned table has to include the partition key, so a plain PRIMARY KEY (id) is off the table. If another table carries a foreign key pointing at that id, you have a migration problem to solve before you start, and it is much better to find that now.

Migrating Without Downtime

The approach that works is to leave the existing table alone and attach it as a partition, which turns a rewrite into a metadata change:

ALTER TABLE events_old
    ADD CONSTRAINT events_old_range
    CHECK (occurred_at >= '2020-01-01' AND occurred_at < '2026-08-01') NOT VALID;

ALTER TABLE events_old VALIDATE CONSTRAINT events_old_range;

ALTER TABLE events ATTACH PARTITION events_old
    FOR VALUES FROM ('2020-01-01') TO ('2026-08-01');

The two-step constraint is the whole trick. Adding it NOT VALID takes a brief lock and skips the scan. VALIDATE CONSTRAINT then does the scan under a weaker lock that readers and writers work around. By the time ATTACH PARTITION runs, Postgres already knows every row satisfies the bound and skips the verification scan that would otherwise hold ACCESS EXCLUSIVE for the length of a full sequential read. Skip the constraint and the attach does that scan itself, with the table locked.

Detaching Beats Deleting

This is the payoff that justifies the whole exercise, and it is easy to undersell. Aging out old data from a monolithic table means DELETE, and a DELETE of ten million rows does not free anything. It writes ten million dead tuples, generates WAL for every one, leaves the indexes bloated, and hands autovacuum a large cleanup job that competes with your live traffic. The table on disk stays exactly as large as it was.

With partitions, the same operation is a catalog change:

ALTER TABLE events DETACH PARTITION events_2025_07 CONCURRENTLY;

No row is touched. No WAL is generated for the data. Nothing needs vacuuming afterwards. The CONCURRENTLY form avoids blocking readers while it happens, and the detached table is now an ordinary standalone table you can archive, dump somewhere cheap, or drop outright. That last distinction matters more than it sounds: detaching is reversible and gives you a table you can still query, while DELETE is a one-way trip that costs far more to perform.

Let pg_partman Run It

Something has to create next month's partition before next month arrives, and the failure mode of forgetting is an insert erroring out at midnight on the first. Hand-rolled cron jobs work until the person who wrote them moves on. pg_partman is the right answer here, and it handles both ends: creating partitions ahead of time and retiring old ones on a policy.

CREATE SCHEMA partman;
CREATE EXTENSION pg_partman WITH SCHEMA partman;

SELECT partman.create_parent(
    p_parent_table := 'public.events',
    p_control      := 'occurred_at',
    p_interval     := '1 month',
    p_premake      := 4
);

Note that pg_partman 5 changed this signature from the 4.x examples still floating around, and p_type now takes range or list rather than the old native. The p_premake value is how many partitions to keep created ahead, so four months of runway on a monthly interval.

Retention is a column in the extension's own config table:

UPDATE partman.part_config
SET retention              = '12 months',
    retention_keep_table   = true
WHERE parent_table = 'public.events';

The default for retention_keep_table is true, which detaches expired partitions and leaves them as standalone tables rather than dropping them. That default is the right one, and it is the same argument as above: you get the space accounting and the maintenance relief immediately, and the data is still sitting there if somebody asks for it next week. Set it to false only when you are certain, or leave it true and have a separate job that dumps and drops detached tables on its own schedule.

Maintenance runs either from the bundled background worker, by adding pg_partman_bgw to shared_preload_libraries, or from a scheduled call:

CALL partman.run_maintenance_proc();

Confirm Pruning Actually Happens

Partitioning silently does nothing if the planner cannot prune, so check rather than assume:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) FROM events
WHERE account_id = 4211
  AND occurred_at >= now() - interval '30 days';

You want a small number of partitions in the plan, and ideally Subplans Removed when the bound is only known at execution time. The thing that bites people is a partition key wrapped in a function or compared against a different type, which defeats pruning with no warning at all. A timestamptz column compared against a timestamp literal is the classic version.

What It Costs

Partition count matters. Planning time grows with the number of partitions, and daily partitions across five years is 1,800 relations the planner considers before it prunes anything. Monthly is usually right for history, weekly if the volume genuinely warrants it. Start coarse, since merging later is harder than splitting.

The payoff is that maintenance becomes proportional to the working set instead of the archive. Vacuum runs against the current partition. The old ones sit frozen and untouched until the day they get detached, which costs nothing.