Category Archives: System Administration

systemd Hardening for Long-Running Daemons

A long-running daemon that talks to the network and a database is the thing an attacker wants most, and it is usually the thing running with the fewest restrictions. systemd ships a large set of sandboxing directives that cost nothing at runtime and are almost never used, mostly because the documentation presents forty options with no indication of which ones matter.

Measure First

systemd will grade the unit for you, which is a better starting point than reading the manual page end to end:

systemd-analyze security tracker.service

You get a per-directive table and an overall exposure score from 0 to 10. A default unit with no hardening scores around 9.6, which the tool cheerfully labels UNSAFE. The score is a rough heuristic rather than a real measure of risk, and it is genuinely useful for ordering the work, since it sorts by exposure and shows what each setting would buy.

The Directives Worth Setting

These are the ones I put on every service, in rough order of value per unit of trouble:

[Service]
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RemoveIPC=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

NoNewPrivileges=yes is the single highest-value line. It blocks setuid escalation for the process and everything it spawns, and almost nothing legitimately needs the alternative.

ProtectSystem=strict mounts the entire filesystem read-only apart from /dev, /proc, and /sys. Combined with ProtectHome=yes, a compromised process cannot write anywhere at all until you grant specific paths back:

ReadWritePaths=/var/lib/tracker /var/log/tracker
RuntimeDirectory=tracker
StateDirectory=tracker

The RuntimeDirectory and StateDirectory forms are better than listing paths by hand, because systemd creates them with the right ownership and cleans up runtime state on stop.

CapabilityBoundingSet= with an empty value drops every capability. Most daemons need none, and the common exception is binding a port below 1024, which needs CAP_NET_BIND_SERVICE added back. Better still, use a socket unit and let systemd hold the privileged socket so the service never needs the capability at all.

RestrictAddressFamilies is worth the thought. A daemon doing DNS and PostgreSQL over TCP needs AF_INET, AF_INET6, and AF_UNIX for the local socket. It has no business creating a raw packet socket or an AF_NETLINK socket, and denying those closes a real avenue.

The Ones That Will Break You

MemoryDenyWriteExecute=yes blocks pages that are both writable and executable. It is excellent against a large class of exploit, and it breaks anything with a JIT. For a C++ daemon it is usually free. For anything embedding a scripting runtime, test carefully.

SystemCallFilter=@system-service is the setting most likely to produce a mysterious failure. The allow list covers what normal services do, and a program using something unusual gets killed with SIGSYS and no useful message. When a service dies immediately after adding it, find the offending call rather than guessing:

journalctl -u tracker.service | grep -i seccomp
auditctl -a always,exit -F arch=b64 -S all -F key=seccomp-debug

ProtectSystem=strict breaks anything writing outside its declared paths, which in practice means log files in unexpected places and PID files in /var/run. Both are better fixed than exempted.

ProtectProc=invisible hides other users’ processes in /proc. Harmless for most daemons and immediately fatal for anything that inspects the process table, including some monitoring agents.

Do It Incrementally

Adding all twenty lines at once and finding the service dead tells you nothing about which line did it. Add the block above minus SystemCallFilter and MemoryDenyWriteExecute, confirm the service still works under real load rather than just starting, then add those two separately.

“Starts successfully” is a weak test. Sandboxing failures show up on code paths that only run occasionally, so the syscall your daemon makes once a day during log rotation is exactly the one the filter blocks. Give a hardened unit a full cycle of whatever periodic work it does before considering it verified.

None of this replaces running as an unprivileged user, keeping the software patched, or putting the database somewhere the internet cannot reach. It is the layer underneath those, and it is roughly twenty lines of configuration that turns a total compromise of the process into a considerably less useful foothold.

PgBouncer, Prepared Statements, and Picking the Right Pool Mode

PgBouncer is the standard answer when an application opens more PostgreSQL connections than the server can usefully service. A process-per-request runtime and a serverless function that dials the database directly get there by completely different routes and land in the same place. The part that goes wrong is the pool mode, because the aggressive setting is the one everybody wants and it silently changes the semantics your code was written against.

Three Modes, One Real Decision

Session pooling assigns a server connection for the life of the client connection. It is safe and it buys you very little, since a worker process holding a connection for the length of a request is the situation you were trying to fix.

Transaction pooling assigns a server connection for the length of a transaction and returns it to the pool at commit. This is the mode that delivers the numbers people install PgBouncer for, and it is the mode with consequences.

Statement pooling returns the connection after every individual statement, which forbids multi-statement transactions entirely. It exists for specific workloads and is almost never what you want.

What Transaction Mode Takes Away

The rule is that anything living outside a transaction stops being reliable, because between two statements you may be on a different backend. That is a longer list than it first appears:

SET / RESET at session scope
LISTEN / NOTIFY
advisory locks taken outside a transaction
WITH HOLD cursors
temporary tables
session-scoped GUCs set by a connection hook

Session-level SET is the one that catches real applications. A framework bootstrap that sets search_path or timezone once on connect works perfectly in development against a direct connection, and in production the setting lands on whichever backend happened to serve that statement. Every later query gets a different backend without it. The failure is intermittent and nearly impossible to reproduce on demand.

Advisory locks are worse, because the failure is silent rather than noisy. A lock taken with pg_advisory_lock() outside a transaction belongs to a session you no longer control. The transaction-scoped variant, pg_advisory_xact_lock(), is released at commit and behaves correctly under transaction pooling. If you use advisory locks for job coordination, that one function name is the difference between working and quietly running the same job twice.

Prepared Statements Are Fixed, with Conditions

For years the answer to prepared statements under transaction pooling was that you could not use them. PgBouncer 1.21, released in October 2023, changed that. Set max_prepared_statements to a non-zero value and PgBouncer tracks named prepared statements itself, rewriting them to internal names and re-preparing them on whichever backend a client lands on:

[pgbouncer]
pool_mode = transaction
max_prepared_statements = 200

The value is the size of an LRU cache of statements kept on each server connection, so it wants to be at least as large as the number of distinct statements a typical request issues. Zero disables the feature and restores the old behavior.

The condition attached is important. This only works for protocol-level prepared statements, the ones sent through the extended query protocol. A literal PREPARE foo AS SELECT ... sent as a plain text query is invisible to PgBouncer and still breaks, because from the outside it is just another statement.

For PHP specifically, this means checking what PDO is actually doing. With ATTR_EMULATE_PREPARES left on, PDO interpolates parameters client-side and sends plain SQL, so there are no real prepared statements and nothing to break. Turn emulation off and you get genuine protocol-level prepares, which is what you want for both correctness and plan reuse, and which is exactly the case max_prepared_statements exists to handle.

$pdo = new PDO($dsn, $user, $pass, [
    PDO::ATTR_EMULATE_PREPARES => false,
    PDO::ATTR_ERRMODE          => PDO::ERRMODE_EXCEPTION,
]);

How I Choose

Start with transaction mode, then go looking for the session state your application depends on rather than waiting for it to surface. Grep for LISTEN, for pg_advisory_lock, for temporary tables, and for anything setting a GUC on connect. Move search_path out of a connection hook and into the connection string, where PgBouncer passes it through as part of the startup parameters and it survives correctly.

Keep a second pool in session mode on a different port for the small number of things that genuinely need a stable session, which is usually a migration runner and whatever handles LISTEN. Two pools with clear rules beats one pool with exceptions nobody remembers.

And size the pool against what PostgreSQL can actually do, since the point of pooling is to keep the database out of the region where it spends more time context switching than working. A pool larger than the database can service just moves the queue.

MTA-STS and TLS-RPT: Enforcing TLS on Inbound Mail

SMTP encryption is opportunistic by default, which means it is optional, which means it is strippable. A sending server connects, looks for STARTTLS in the EHLO response, and upgrades if it sees it. Remove that one line in transit and the sender falls back to plaintext without complaint, because it has no way to know TLS was ever supposed to happen. MTA-STS is how you tell it.

Why STARTTLS Alone Is Not Enough

The gap is that opportunistic TLS has no expectation to violate. A sender that fails to negotiate TLS has no basis to refuse delivery, since plenty of legitimate mail servers still do not offer it. Certificate validation is usually skipped for the same reason: a mail server presenting a self-signed or expired certificate is common enough that treating it as fatal would break real delivery.

So the default posture is an encrypted channel to an unverified party, downgradeable by anyone in the path. MTA-STS, specified in RFC 8461, lets a receiving domain publish a policy saying TLS is required, the certificate must validate, and here are the hostnames allowed to receive mail. Senders that implement it will refuse to deliver rather than fall back.

The Three Pieces

First, a DNS TXT record at the _mta-sts label telling senders a policy exists:

_mta-sts.example.com.  IN  TXT  "v=STSv1; id=20260916120000Z;"

The id is the whole mechanism for cache invalidation. Senders compare it against the one they saw last time, and they only re-fetch the policy when it changes. Bump it on every policy edit or your change will not be noticed until caches expire on their own.

Second, the policy itself, served over HTTPS from a specific host and path:

https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 604800

mode takes enforce, testing, or none. Every MX that can receive mail for the domain needs a line, and a wildcard like *.example.com matches only the leftmost label, so it covers mail.example.com and not foo.bar.example.com. max_age is in seconds with a ceiling of 31557600, about a year.

Third, and this is the part people trip over: the mta-sts host serving that file needs a valid, publicly trusted certificate of its own. The policy is only as trustworthy as the HTTPS connection that delivered it, so a sender that cannot validate that certificate discards the policy entirely. You have now made your mail delivery depend on a certificate on a host that has nothing else to do with mail, which is exactly the kind of endpoint that quietly expires.

Start in Testing Mode

Publishing mode: enforce as the first step is how you find out about your forgotten backup MX by having mail to it rejected. mode: testing makes senders evaluate the policy, report failures, and deliver anyway.

Leave it there long enough to see a full cycle of your real traffic. A week is reasonable, longer if you have partners who send in bursts. The reports are the point of the exercise, which brings up the other half.

TLS-RPT Tells You What Broke

RFC 8460 defines a companion record that asks senders to report what happened. It is cheap to add and it is the only feedback channel you get:

_smtp._tls.example.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:tlsrpt@example.com"

Reports arrive as JSON, daily, from each sending organization that implements it. The useful part is the failure detail:

{
  "organization-name": "Example Sender Inc",
  "date-range": { "start-datetime": "2026-09-16T00:00:00Z",
                  "end-datetime":   "2026-09-16T23:59:59Z" },
  "report-id": "2026-09-16T00:00:00Z_example.com",
  "policies": [{
    "policy": { "policy-type": "sts", "policy-domain": "example.com" },
    "summary": { "total-successful-session-count": 8214,
                 "total-failure-session-count": 17 },
    "failure-details": [{
      "result-type": "certificate-host-mismatch",
      "sending-mta-ip": "203.0.113.9",
      "receiving-mx-hostname": "mail2.example.com",
      "failed-session-count": 17
    }]
  }]
}

The result-type values are specific enough to act on directly. starttls-not-supported, certificate-expired, certificate-host-mismatch, certificate-not-trusted, validation-failure, and on the policy side sts-policy-fetch-error, sts-policy-invalid, and sts-webpki-invalid. That last group points at your policy host rather than your mail servers, which is a distinction worth internalizing before you start debugging the wrong machine.

The Ways This Bites

Caching cuts both ways. Once a sender has your policy with a week-long max_age, a mistake persists for that long even after you fix the file, unless the sender happens to re-check the id. Adding an MX means updating the policy and bumping the id before the new host starts receiving, and doing it in that order.

The certificate on the mta-sts web host is the failure nobody plans for. It is usually a static file on a server that gets less attention than the mail infrastructure, and when it expires, compliant senders stop trusting your policy. Whether that means mail is deferred or the policy is simply ignored depends on the sender.

Your MX certificates now genuinely have to be correct as well. Names have to match what the policy lists, chains have to be complete, and expiry stops being cosmetic. That is the point of turning it on, and it is worth knowing before you flip to enforce.

To check what you have published, the MTA-STS checker on mrdns.com fetches the record and the policy together and tells you whether they agree. For the mail server configuration underneath it, goodtls.com has per-application guides covering Postfix, Exim, Dovecot, and the rest.

Partitioning a Table That Never Stops Growing

Most systems I have worked on end up with a handful of tables that only ever grow. Event history, audit trails, request logs, anything append-only. None of them are interesting individually, and together they eventually dominate the maintenance budget: vacuum takes longer every month, index rebuilds turn into scheduled events, and a query that only wants last week has to reason about years of rows to prove it.

Declarative partitioning is the fix, and PostgreSQL has been good at it since 12. The parts worth writing down are how to get an existing table into it without downtime, and how to stop hand-managing it afterwards.

Partition on the Column You Actually Filter

Range partitioning by time is the obvious choice for append-only data, and the partition key has to appear in your WHERE clauses or you gain nothing. Partition by occurred_at while every query filters on account_id and each one still touches every partition.

CREATE TABLE events (
    id bigint GENERATED ALWAYS AS IDENTITY,
    account_id integer NOT NULL,
    occurred_at timestamptz NOT NULL,
    event_type smallint NOT NULL,
    PRIMARY KEY (id, occurred_at)
) PARTITION BY RANGE (occurred_at);

Note the primary key. Every unique constraint on a partitioned table has to include the partition key, so a plain PRIMARY KEY (id) is off the table. If another table carries a foreign key pointing at that id, you have a migration problem to solve before you start, and it is much better to find that now.

Migrating Without Downtime

The approach that works is to leave the existing table alone and attach it as a partition, which turns a rewrite into a metadata change:

ALTER TABLE events_old
    ADD CONSTRAINT events_old_range
    CHECK (occurred_at >= '2020-01-01' AND occurred_at < '2026-08-01') NOT VALID;

ALTER TABLE events_old VALIDATE CONSTRAINT events_old_range;

ALTER TABLE events ATTACH PARTITION events_old
    FOR VALUES FROM ('2020-01-01') TO ('2026-08-01');

The two-step constraint is the whole trick. Adding it NOT VALID takes a brief lock and skips the scan. VALIDATE CONSTRAINT then does the scan under a weaker lock that readers and writers work around. By the time ATTACH PARTITION runs, Postgres already knows every row satisfies the bound and skips the verification scan that would otherwise hold ACCESS EXCLUSIVE for the length of a full sequential read. Skip the constraint and the attach does that scan itself, with the table locked.

Detaching Beats Deleting

This is the payoff that justifies the whole exercise, and it is easy to undersell. Aging out old data from a monolithic table means DELETE, and a DELETE of ten million rows does not free anything. It writes ten million dead tuples, generates WAL for every one, leaves the indexes bloated, and hands autovacuum a large cleanup job that competes with your live traffic. The table on disk stays exactly as large as it was.

With partitions, the same operation is a catalog change:

ALTER TABLE events DETACH PARTITION events_2025_07 CONCURRENTLY;

No row is touched. No WAL is generated for the data. Nothing needs vacuuming afterwards. The CONCURRENTLY form avoids blocking readers while it happens, and the detached table is now an ordinary standalone table you can archive, dump somewhere cheap, or drop outright. That last distinction matters more than it sounds: detaching is reversible and gives you a table you can still query, while DELETE is a one-way trip that costs far more to perform.

Let pg_partman Run It

Something has to create next month's partition before next month arrives, and the failure mode of forgetting is an insert erroring out at midnight on the first. Hand-rolled cron jobs work until the person who wrote them moves on. pg_partman is the right answer here, and it handles both ends: creating partitions ahead of time and retiring old ones on a policy.

CREATE SCHEMA partman;
CREATE EXTENSION pg_partman WITH SCHEMA partman;

SELECT partman.create_parent(
    p_parent_table := 'public.events',
    p_control      := 'occurred_at',
    p_interval     := '1 month',
    p_premake      := 4
);

Note that pg_partman 5 changed this signature from the 4.x examples still floating around, and p_type now takes range or list rather than the old native. The p_premake value is how many partitions to keep created ahead, so four months of runway on a monthly interval.

Retention is a column in the extension's own config table:

UPDATE partman.part_config
SET retention              = '12 months',
    retention_keep_table   = true
WHERE parent_table = 'public.events';

The default for retention_keep_table is true, which detaches expired partitions and leaves them as standalone tables rather than dropping them. That default is the right one, and it is the same argument as above: you get the space accounting and the maintenance relief immediately, and the data is still sitting there if somebody asks for it next week. Set it to false only when you are certain, or leave it true and have a separate job that dumps and drops detached tables on its own schedule.

Maintenance runs either from the bundled background worker, by adding pg_partman_bgw to shared_preload_libraries, or from a scheduled call:

CALL partman.run_maintenance_proc();

Confirm Pruning Actually Happens

Partitioning silently does nothing if the planner cannot prune, so check rather than assume:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) FROM events
WHERE account_id = 4211
  AND occurred_at >= now() - interval '30 days';

You want a small number of partitions in the plan, and ideally Subplans Removed when the bound is only known at execution time. The thing that bites people is a partition key wrapped in a function or compared against a different type, which defeats pruning with no warning at all. A timestamptz column compared against a timestamp literal is the classic version.

What It Costs

Partition count matters. Planning time grows with the number of partitions, and daily partitions across five years is 1,800 relations the planner considers before it prunes anything. Monthly is usually right for history, weekly if the volume genuinely warrants it. Start coarse, since merging later is harder than splitting.

The payoff is that maintenance becomes proportional to the working set instead of the archive. Vacuum runs against the current partition. The old ones sit frozen and untouched until the day they get detached, which costs nothing.

What Migrating My AWS Fleet to Graviton Taught Me

Over the last while I have moved everything I run on AWS to Graviton, the ARM CPUs AWS designs in house: the EC2 instances behind my services and the ECS containers that make up most of the actual workload. The short version is that it was worth it, the bill went down, and almost nothing about how I write or run code had to change. The longer version has a few sharp edges worth writing down, plus some genuinely interesting silicon arriving now in Graviton5.

Why ARM in the Cloud

The pitch is not “ARM is faster,” it is price/performance. For the same dollar you generally get more useful work out of a Graviton instance than the equivalent Intel or AMD one, and AWS prices the instances lower on top of that. Set expectations correctly, though: a single Graviton core will not smoke a top-bin x86 core on a single-threaded benchmark. The win shows up across a fleet of web servers, API workers, queue consumers, and databases: more cores per dollar and steady throughput under real concurrent load. If you are chasing the lowest latency on one hot thread, measure first.

ARM vs Intel vs AMD in Practice

  • Graviton is my default: best price/performance for general compute, web tiers, microservices, and most databases, at the lowest cost and power.
  • AMD is the value x86 option when something genuinely needs x86 but not Intel specifically.
  • Intel is a deliberate choice for AVX-512, Intel-specific acceleration, or vendors that only certify on Intel. It is the priciest of the three.

What actually decides it is rarely the chip and almost always the software. If everything you run is open source or built from your own source, ARM is a non issue. The moment a proprietary, x86 only binary shows up, that workload stays on x86 until the vendor ships an ARM build. So the real question per service is not “is ARM fast enough,” it is “does every binary in this container have an arm64 version.”

What Actually Bit Me

None of these were dealbreakers, but I wish I had batched them up front instead of finding them one service at a time.

The first one is the biggest mindset shift: your images must be multi-arch. Your base images, every layer, and your own build all need linux/arm64. I moved to docker buildx and publish multi-arch manifests so one tag resolves correctly wherever it runs:

docker buildx build \
    --platform linux/amd64,linux/arm64 \
    -t myrepo/myservice:latest --push .

Then ECS has to be told which architecture to use. The task definition’s runtimePlatform defaults to x86, so set it to ARM64 and run on Graviton capacity or Fargate:

"runtimePlatform": {
    "cpuArchitecture": "ARM64",
    "operatingSystemFamily": "LINUX"
}

When a service misbehaves after the switch, the first thing I check is whether the published tag actually contains both architectures. This one-liner saves a lot of guessing:

docker buildx imagetools inspect myrepo/myservice:latest
# look for linux/amd64 and linux/arm64 in the platform list

The rest are smaller traps worth knowing up front:

  • Emulated CI is slow. Building arm64 on an x86 runner means QEMU, which is painful for anything compiled. You no longer have to: GitHub-hosted native arm64 runners are generally available, free for public repositories and now offered for private ones too, so the pipeline builds on real ARM hardware instead of emulating it.
  • Native dependencies are the long tail. Interpreted code moves for free; compiled extensions (native npm modules, Python wheels, cgo, old vendored binaries) each need an arm64 build.
  • Check your sidecars. Logging, metrics, and security agents need arm64 images too. The mainstream ones have them, but a sidecar that silently fails to start will ruin your afternoon.
  • Managed services are the easy win. RDS, ElastiCache, and Lambda offer Graviton with no code change. Start there to build confidence.

Graviton5: What Changed

While I was mid migration, AWS moved the goalposts. The lineage shows the trajectory:

Gen Cores Microarch ISA Clock Memory
Graviton2 64 Neoverse N1 Armv8.2-A 2.5 GHz DDR4
Graviton3 64 Neoverse V1 Armv8.4-A 2.6 GHz DDR5-4800
Graviton4 96 Neoverse V2 Armv9.0-A 2.8 GHz DDR5-5600
Graviton5 192 Neoverse V3 Armv9.2-A 3.3 GHz DDR5-8800

Graviton5, announced in December 2025 with M9g instances reaching general availability in mid 2026, is a bigger step than the version bump suggests. It doubles Graviton4 to 192 Neoverse V3 cores, built on a 3nm process as four chiplets of 48 cores each with up to ~420 GB/s between chiplets. It carries 192 MB of L3 cache (around five times the prior generation), 2 MB of L2 per core, DDR5-8800, and PCIe Gen6. It also ships a new Nitro Isolation Engine, which AWS describes as a formally verified hypervisor: the VM isolation property is mathematically proven, not just tested.

AWS quotes roughly 25% better compute than Graviton4 per core, with up to ~35% on web apps, up to ~35% on ML inference, and ~30% on databases from improved branch prediction. Take vendor percentages with the usual grain of salt and benchmark your own workload, but the architecture behind them is real. For densely packed container fleets the per core number matters less than doubling the cores while growing cache and memory bandwidth. The general purpose M9g and M9gd are out now; the compute optimized C9g and memory optimized R9g are the variants I am waiting on, since most of my fleet maps to those shapes.

Is It Worth It

For me, clearly yes. If your stack is containers and managed services built from source or mainstream images, the migration is mostly mechanical: make images multi-arch, set the architecture on your tasks, fix the native dependency stragglers. Start with managed services for an easy win, then move containers one service at a time so you can roll back cleanly. The one piece of advice I would give my past self: audit dependencies and sidecars for arm64 support before flipping anything. Ninety percent of the work is trivial. The other ten percent is one unmaintained binary you forgot about, and it is much nicer to find it on a spreadsheet than in a failing deployment.