Partitioning a Table That Never Stops Growing

Most systems I have worked on end up with a handful of tables that only ever grow. Event history, audit trails, request logs, anything append-only. None of them are interesting individually, and together they eventually dominate the maintenance budget: vacuum takes longer every month, index rebuilds turn into scheduled events, and a query that only wants last week has to reason about years of rows to prove it.

Declarative partitioning is the fix, and PostgreSQL has been good at it since 12. The parts worth writing down are how to get an existing table into it without downtime, and how to stop hand-managing it afterwards.

Partition on the Column You Actually Filter

Range partitioning by time is the obvious choice for append-only data, and the partition key has to appear in your WHERE clauses or you gain nothing. Partition by occurred_at while every query filters on account_id and each one still touches every partition.

CREATE TABLE events (
    id bigint GENERATED ALWAYS AS IDENTITY,
    account_id integer NOT NULL,
    occurred_at timestamptz NOT NULL,
    event_type smallint NOT NULL,
    PRIMARY KEY (id, occurred_at)
) PARTITION BY RANGE (occurred_at);

Note the primary key. Every unique constraint on a partitioned table has to include the partition key, so a plain PRIMARY KEY (id) is off the table. If another table carries a foreign key pointing at that id, you have a migration problem to solve before you start, and it is much better to find that now.

Migrating Without Downtime

The approach that works is to leave the existing table alone and attach it as a partition, which turns a rewrite into a metadata change:

ALTER TABLE events_old
    ADD CONSTRAINT events_old_range
    CHECK (occurred_at >= '2020-01-01' AND occurred_at < '2026-08-01') NOT VALID;

ALTER TABLE events_old VALIDATE CONSTRAINT events_old_range;

ALTER TABLE events ATTACH PARTITION events_old
    FOR VALUES FROM ('2020-01-01') TO ('2026-08-01');

The two-step constraint is the whole trick. Adding it NOT VALID takes a brief lock and skips the scan. VALIDATE CONSTRAINT then does the scan under a weaker lock that readers and writers work around. By the time ATTACH PARTITION runs, Postgres already knows every row satisfies the bound and skips the verification scan that would otherwise hold ACCESS EXCLUSIVE for the length of a full sequential read. Skip the constraint and the attach does that scan itself, with the table locked.

Detaching Beats Deleting

This is the payoff that justifies the whole exercise, and it is easy to undersell. Aging out old data from a monolithic table means DELETE, and a DELETE of ten million rows does not free anything. It writes ten million dead tuples, generates WAL for every one, leaves the indexes bloated, and hands autovacuum a large cleanup job that competes with your live traffic. The table on disk stays exactly as large as it was.

With partitions, the same operation is a catalog change:

ALTER TABLE events DETACH PARTITION events_2025_07 CONCURRENTLY;

No row is touched. No WAL is generated for the data. Nothing needs vacuuming afterwards. The CONCURRENTLY form avoids blocking readers while it happens, and the detached table is now an ordinary standalone table you can archive, dump somewhere cheap, or drop outright. That last distinction matters more than it sounds: detaching is reversible and gives you a table you can still query, while DELETE is a one-way trip that costs far more to perform.

Let pg_partman Run It

Something has to create next month's partition before next month arrives, and the failure mode of forgetting is an insert erroring out at midnight on the first. Hand-rolled cron jobs work until the person who wrote them moves on. pg_partman is the right answer here, and it handles both ends: creating partitions ahead of time and retiring old ones on a policy.

CREATE SCHEMA partman;
CREATE EXTENSION pg_partman WITH SCHEMA partman;

SELECT partman.create_parent(
    p_parent_table := 'public.events',
    p_control      := 'occurred_at',
    p_interval     := '1 month',
    p_premake      := 4
);

Note that pg_partman 5 changed this signature from the 4.x examples still floating around, and p_type now takes range or list rather than the old native. The p_premake value is how many partitions to keep created ahead, so four months of runway on a monthly interval.

Retention is a column in the extension's own config table:

UPDATE partman.part_config
SET retention              = '12 months',
    retention_keep_table   = true
WHERE parent_table = 'public.events';

The default for retention_keep_table is true, which detaches expired partitions and leaves them as standalone tables rather than dropping them. That default is the right one, and it is the same argument as above: you get the space accounting and the maintenance relief immediately, and the data is still sitting there if somebody asks for it next week. Set it to false only when you are certain, or leave it true and have a separate job that dumps and drops detached tables on its own schedule.

Maintenance runs either from the bundled background worker, by adding pg_partman_bgw to shared_preload_libraries, or from a scheduled call:

CALL partman.run_maintenance_proc();

Confirm Pruning Actually Happens

Partitioning silently does nothing if the planner cannot prune, so check rather than assume:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) FROM events
WHERE account_id = 4211
  AND occurred_at >= now() - interval '30 days';

You want a small number of partitions in the plan, and ideally Subplans Removed when the bound is only known at execution time. The thing that bites people is a partition key wrapped in a function or compared against a different type, which defeats pruning with no warning at all. A timestamptz column compared against a timestamp literal is the classic version.

What It Costs

Partition count matters. Planning time grows with the number of partitions, and daily partitions across five years is 1,800 relations the planner considers before it prunes anything. Monthly is usually right for history, weekly if the volume genuinely warrants it. Start coarse, since merging later is harder than splitting.

The payoff is that maintenance becomes proportional to the working set instead of the archive. Vacuum runs against the current partition. The old ones sit frozen and untouched until the day they get detached, which costs nothing.

PostgreSQL 19 Beta: Property Graphs and Online REPACK

PostgreSQL 19 is at Beta 2, with the final release expected around September or October, so syntax and behavior can still shift. Two things in this cycle are worth the upgrade on their own: SQL/PGQ property graphs, and the new REPACK command.

Property Graphs Behave Like Views

SQL/PGQ comes from SQL:2023 (ISO/IEC 9075-16), and CREATE PROPERTY GRAPH materializes nothing. It is a new relkind that acts like a view: the rewriter turns your graph pattern into a plain relational query against the underlying tables, using the indexes you already have.

Take a small asset inventory: domains, the hostnames they publish, and the addresses those hostnames resolve to.

CREATE TABLE domains (
    domain_id integer PRIMARY KEY,
    name text NOT NULL
);

CREATE TABLE hosts (
    host_id integer PRIMARY KEY,
    hostname text NOT NULL
);

CREATE TABLE addresses (
    address_id integer PRIMARY KEY,
    ip inet NOT NULL
);

CREATE TABLE domain_hosts (
    domain_hosts_id integer PRIMARY KEY,
    domain_id integer REFERENCES domains (domain_id),
    host_id integer REFERENCES hosts (host_id),
    record_type text NOT NULL
);

CREATE TABLE host_addresses (
    host_addresses_id integer PRIMARY KEY,
    host_id integer REFERENCES hosts (host_id),
    address_id integer REFERENCES addresses (address_id)
);

Entity tables become vertices, join tables become edges, and the definition stays short because the keys already describe how everything connects:

CREATE PROPERTY GRAPH infra
    VERTEX TABLES (
        domains   LABEL domain,
        hosts     LABEL host,
        addresses LABEL address
    )
    EDGE TABLES (
        domain_hosts   SOURCE domains DESTINATION hosts     LABEL resolves_to,
        host_addresses SOURCE hosts   DESTINATION addresses LABEL has_address
    );

Without those keys you spell them out with KEY (...), SOURCE KEY (...) REFERENCES ..., and DESTINATION KEY (...) REFERENCES .... Queries go through GRAPH_TABLE, which sits in the FROM clause and acts like a table function:

SELECT domain_name, hostname
FROM GRAPH_TABLE (infra
    MATCH (d IS domain)-[IS resolves_to]->(h IS host)
          -[IS has_address]->(a IS address WHERE a.ip = inet '203.0.113.10')
    COLUMNS (d.name AS domain_name, h.hostname AS hostname)
)
ORDER BY domain_name;

The same question the old way, which is what the rewriter produces anyway:

SELECT d.name, h.hostname
FROM domains d
    JOIN domain_hosts dh ON dh.domain_id = d.domain_id
    JOIN hosts h ON h.host_id = dh.host_id
    JOIN host_addresses ha ON ha.host_id = h.host_id
    JOIN addresses a ON a.address_id = ha.address_id
WHERE a.ip = inet '203.0.113.10'
ORDER BY d.name;

At two hops it is a wash. The graph version stops growing at four, while the join version keeps piling up aliases and ON clauses that all have to stay correct.

The Pattern Language

A vertex is (), an edge is -[]->, and a path alternates between them. Labels match with IS, and | means “or”. Edges are directed, so <-[ ]- goes the other way and -[ ]- matches either. Drop the brackets when the edge needs no filter of its own:

(IS host)-[IS has_address]->(IS address|cidr_block)
(IS address)<-[IS has_address]-(IS host)
(IS domain)->(IS host)->(IS address)

Bind a variable and you can pull its properties in COLUMNS, including columns off the edge table itself:

MATCH (d IS domain)-[r IS resolves_to]->(h IS host)
COLUMNS (d.name AS domain_name, r.record_type AS via, h.hostname AS hostname)

Now the part that decides whether this is useful to you. PostgreSQL 19 implements the core of SQL/PGQ and leaves most optional subfeatures out, and the omissions cluster in one place: variable-depth traversal. The conformance appendix lists quantified paths (G035), quantified edges (G036), bounded and unbounded quantifiers (G060, G061), path variables (G004), and shortest-path search (G017, G018) as unsupported. Label conjunction (G071), negation (G072), and wildcards (G074) are out too, leaving | as your only operator.

So every hop gets written out. You cannot ask for “anything reachable within three hops” or “the shortest path between these two vertices”, which is what most people mean when they say graph query. Transitive closure and reachability still belong to recursive CTEs. Fixed-depth traversal over a known shape is what this first cut does.

Two more edges to watch: a path cannot begin or end with an edge pattern, and two edge patterns cannot sit next to each other. In psql, \dG lists your graphs.

REPACK Replaces VACUUM FULL and CLUSTER

VACUUM FULL and CLUSTER have always done nearly the same job under two names that explain neither. PostgreSQL 19 unifies them into REPACK, keeping the old commands for compatibility.

REPACK [ ( option [, ...] ) ] [ table_and_columns [ USING INDEX [ index_name ] ] ]
REPACK [ ( option [, ...] ) ] USING INDEX

where option can be one of:

    VERBOSE [ boolean ]
    ANALYZE [ boolean ]
    CONCURRENTLY [ boolean ]

Plain REPACK rewrites the table with no free space beyond what fillfactor reserves, which is the old VACUUM FULL. Add USING INDEX and rows come out physically ordered by that index, which is the old CLUSTER.

REPACK listings;
REPACK listings USING INDEX listings_host_date_idx;
REPACK (ANALYZE, VERBOSE) listings USING INDEX listings_host_date_idx;

Progress lands in the new pg_stat_progress_repack view. With no table name, REPACK processes every table and materialized view you hold MAINTAIN on, which is a fine way to ruin an afternoon on a production cluster.

CONCURRENTLY Is the Reason to Care

Plain REPACK holds ACCESS EXCLUSIVE on the whole table for the duration, so on anything large it means a maintenance window. CONCURRENTLY changes that: Postgres builds the new heap and indexes while the table stays readable and writable, captures concurrent DML through logical decoding, replays it, and only then takes ACCESS EXCLUSIVE long enough to swap the files in.

REPACK (CONCURRENTLY) listings USING INDEX listings_host_date_idx;

That is what pg_repack and pg_squeeze have done as extensions for years, now in core with no triggers and no external binary. On a managed provider that will not install your extension of choice, that is the whole ballgame.

CONCURRENTLY is rejected when:

  • the table is UNLOGGED
  • the table is partitioned
  • the table has neither a primary key nor an index-based replica identity
  • the target is a system catalog or a TOAST table
  • the command is inside a transaction block
  • max_repack_replication_slots has no slot available

That last one is a new GUC, default 5, settable only at server start, and each concurrent repack holds a slot for its whole run. Budget it against your other slots. Temporary disk is the other cost, since DML landing during the copy is buffered to a temp file until it can be applied, so give maintenance_work_mem some room first.

The caveat worth putting in your runbook is the MVCC one, which does not apply to the non-concurrent form:

Some commands, currently only TRUNCATE, the table-rewriting forms of ALTER TABLE and REPACK with the CONCURRENTLY option, are not MVCC-safe. This means that after the truncation or rewrite commits, the table will appear empty to concurrent transactions, if they are using a snapshot taken before the command committed.

Anything that already touched the table holds an ACCESS SHARE lock and blocks the swap, so it is safe. The exposure is a long repeatable-read or serializable transaction that took its snapshot early, has not read the table yet, and reads it after the swap commits. It sees an empty table.

Other Changes Worth Knowing

A short pass over the rest of the release that will actually touch you:

  • JIT is off by default. The jit parameter now defaults to off, on the grounds that the cost-based triggering was unreliable. If you had it doing useful work, turn it back on deliberately.
  • TOAST compression defaults to lz4 where the build has it, falling back to pglz otherwise. Cheaper compression and decompression on wide text and JSON columns, for the cost of slightly larger stored values.
  • Autovacuum goes parallel. Workers are capped by autovacuum_max_parallel_workers, with a per-table autovacuum_parallel_workers storage parameter. A new scoring system decides which tables get processed first, tunable through autovacuum_vacuum_score_weight, autovacuum_freeze_score_weight, and three siblings.
  • Planner hints, more or less. The new pg_plan_advice module adds a mini-language for pinning planner decisions: JOIN_ORDER(), HASH_JOIN(), SEQ_SCAN(), NO_GATHER() and others. Run EXPLAIN (PLAN_ADVICE) on a plan you like, then feed the generated advice back. pg_stash_advice stores and applies it per query ID.
  • Upgrade landmines. RADIUS authentication over UDP is gone, successful MD5 authentication now logs a warning, max_locks_per_transaction doubles to 128, and standard_conforming_strings is forced on, so old dumps taken with it off will not load. pg_upgrade also refuses clusters holding btree_gist indexes on inet or cidr, whose default opclass changed to fix a correctness bug.

Worth the Upgrade

REPACK CONCURRENTLY is the one I will use in week one, since bloat on a high-churn table has always cost either an outage window or an extension the platform team has to bless. Property graphs I am more measured about: clean implementation, free when unused, and a two-hop pattern does beat five joins. Until the quantifiers land it will not displace a recursive CTE, and it is certainly not a substitute for a real graph database if you traverse at depth. What it does is give the SQL:2023 syntax a home so the follow-up patches have somewhere to land.

Generator Labs Nagios and Zabbix Plugins for Blacklist and Certificate Monitoring

Generator Labs Nagios plugin GitHub repository

If you run Nagios or Zabbix, Generator Labs monitoring data can now flow directly into your existing infrastructure monitoring stack. Updated plugins for both platforms are available on GitHub, adding support for certificate monitoring alongside the existing blacklist monitoring checks.

Nagios Plugin

The Nagios plugin is a bash script that calls the Generator Labs API and maps the response to standard Nagios exit codes:

  • OK when no issues are detected
  • CRITICAL when active listings or certificate errors are found
  • UNKNOWN on API or configuration errors

Two check types are supported:

  • rbl: checks for active blacklist listings on a host
  • cert: checks for active certificate errors on a monitor

Install by copying check_generator.sh to your Nagios plugins directory and adding the command and service definitions. A complete example configuration is included in the repository.

Zabbix Plugin

The Zabbix plugin uses the same API and exposes the same check types as Zabbix external checks. Import the provided template, set your Account SID and API token as macros, and hosts are automatically discovered and mapped to Zabbix items and triggers.

Both plugins replace the legacy RBLTracker plugins. If you were running the old versions, remove them and install the updated ones. The check syntax is unchanged, so existing service definitions don’t need to be updated.

Documentation

Query Your Generator Labs Monitoring Data from Any AI Assistant

Generator Labs MCP server documentation page

Generator Labs now runs a hosted MCP (Model Context Protocol) server, which means any MCP-aware AI tool can read your monitoring data and run on-demand checks directly from the chat interface. No switching tabs, no copying host names, no manual lookups.

What You Can Ask

Once connected, your AI assistant has access to your full account data and can answer questions like:

  • “Which of my hosts are currently listed on any RBL?”
  • “Show me certificates expiring in the next 30 days.”
  • “Run a check on mail.example.com and tell me what flagged it.”
  • “What alerts went out this week, and to which contacts?”

The AI translates your request into tool calls, returns results in plain language, and can chain follow-up queries without leaving the conversation.

Supported Tools

Area What’s Available
Blacklist Monitoring List and inspect hosts, active listings, profiles, check history, run manual checks
Certificate Monitoring List monitors, view expiring certs, inspect errors, run compliance audits
Notifications View contacts, groups, webhooks, and recent alerts
Account Summary, balance, and server health

Connecting

The MCP endpoint is at https://api.generatorlabs.com/4.0/mcp. For Claude Desktop, add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "generator-labs": {
      "type": "http",
      "url": "https://api.generatorlabs.com/4.0/mcp",
      "headers": {
        "Authorization": "Basic <base64 of AccountSID:AuthToken>"
      }
    }
  }
}

For Claude.ai, ChatGPT, and other tools that support OAuth 2.1, add a custom connector with the endpoint URL and complete the browser-based auth flow.

Full setup instructions are in the MCP documentation.

Monitor Your Email Reputation and Certificates in Prometheus

Generator Labs Prometheus exporter GitHub repository

If your team already runs Prometheus, you can now pull Generator Labs monitoring data directly into your metrics stack. The Generator Labs Prometheus exporter exposes blacklist listing status and SSL certificate expiry as standard Prometheus metrics, making it straightforward to build Grafana dashboards or set up alerting rules alongside the rest of your infrastructure.

What It Exports

The exporter surfaces metrics for both products:

  • Blacklist monitoring: active listing status per host, listing counts by source type, last check timestamps
  • Certificate monitoring: days until expiration per monitor, active error status, chain and hostname validation results

These map cleanly to Grafana panels: a certificate expiry countdown per domain, a listing status heatmap across your host inventory, or a single alert rule that fires when any host gets listed or any cert drops below 14 days.

Installation

Three options are available depending on your environment:

Pre-built binary: download from the GitHub releases page and run directly. No dependencies.

Docker:

docker run -e GENERATOR_LABS_ACCOUNT_SID=your_sid \
           -e GENERATOR_LABS_AUTH_TOKEN=your_token \
           -p 9090:9090 \
           ghcr.io/generator-labs/prometheus-exporter:latest

Build from source: requires Go 1.21 or later.

Configuration

The exporter takes two credentials: your Account SID and API token from your Generator Labs account settings. Supply them as flags (--account-sid, --auth-token) or the environment variables above. The metrics endpoint is exposed on port 9090 by default.

Add a scrape config to your prometheus.yml:

scrape_configs:
  - job_name: 'generator-labs'
    static_configs:
      - targets: ['localhost:9090']

Full setup guide on GitHub.