Tag Archives: PHP

PgBouncer, Prepared Statements, and Picking the Right Pool Mode

PgBouncer is the standard answer when an application opens more PostgreSQL connections than the server can usefully service. A process-per-request runtime and a serverless function that dials the database directly get there by completely different routes and land in the same place. The part that goes wrong is the pool mode, because the aggressive setting is the one everybody wants and it silently changes the semantics your code was written against.

Three Modes, One Real Decision

Session pooling assigns a server connection for the life of the client connection. It is safe and it buys you very little, since a worker process holding a connection for the length of a request is the situation you were trying to fix.

Transaction pooling assigns a server connection for the length of a transaction and returns it to the pool at commit. This is the mode that delivers the numbers people install PgBouncer for, and it is the mode with consequences.

Statement pooling returns the connection after every individual statement, which forbids multi-statement transactions entirely. It exists for specific workloads and is almost never what you want.

What Transaction Mode Takes Away

The rule is that anything living outside a transaction stops being reliable, because between two statements you may be on a different backend. That is a longer list than it first appears:

SET / RESET at session scope
LISTEN / NOTIFY
advisory locks taken outside a transaction
WITH HOLD cursors
temporary tables
session-scoped GUCs set by a connection hook

Session-level SET is the one that catches real applications. A framework bootstrap that sets search_path or timezone once on connect works perfectly in development against a direct connection, and in production the setting lands on whichever backend happened to serve that statement. Every later query gets a different backend without it. The failure is intermittent and nearly impossible to reproduce on demand.

Advisory locks are worse, because the failure is silent rather than noisy. A lock taken with pg_advisory_lock() outside a transaction belongs to a session you no longer control. The transaction-scoped variant, pg_advisory_xact_lock(), is released at commit and behaves correctly under transaction pooling. If you use advisory locks for job coordination, that one function name is the difference between working and quietly running the same job twice.

Prepared Statements Are Fixed, with Conditions

For years the answer to prepared statements under transaction pooling was that you could not use them. PgBouncer 1.21, released in October 2023, changed that. Set max_prepared_statements to a non-zero value and PgBouncer tracks named prepared statements itself, rewriting them to internal names and re-preparing them on whichever backend a client lands on:

[pgbouncer]
pool_mode = transaction
max_prepared_statements = 200

The value is the size of an LRU cache of statements kept on each server connection, so it wants to be at least as large as the number of distinct statements a typical request issues. Zero disables the feature and restores the old behavior.

The condition attached is important. This only works for protocol-level prepared statements, the ones sent through the extended query protocol. A literal PREPARE foo AS SELECT ... sent as a plain text query is invisible to PgBouncer and still breaks, because from the outside it is just another statement.

For PHP specifically, this means checking what PDO is actually doing. With ATTR_EMULATE_PREPARES left on, PDO interpolates parameters client-side and sends plain SQL, so there are no real prepared statements and nothing to break. Turn emulation off and you get genuine protocol-level prepares, which is what you want for both correctness and plan reuse, and which is exactly the case max_prepared_statements exists to handle.

$pdo = new PDO($dsn, $user, $pass, [
    PDO::ATTR_EMULATE_PREPARES => false,
    PDO::ATTR_ERRMODE          => PDO::ERRMODE_EXCEPTION,
]);

How I Choose

Start with transaction mode, then go looking for the session state your application depends on rather than waiting for it to surface. Grep for LISTEN, for pg_advisory_lock, for temporary tables, and for anything setting a GUC on connect. Move search_path out of a connection hook and into the connection string, where PgBouncer passes it through as part of the startup parameters and it survives correctly.

Keep a second pool in session mode on a different port for the small number of things that genuinely need a stable session, which is usually a migration runner and whatever handles LISTEN. Two pools with clear rules beats one pool with exceptions nobody remembers.

And size the pool against what PostgreSQL can actually do, since the point of pooling is to keep the database out of the region where it spends more time context switching than working. A pool larger than the database can service just moves the queue.

Net_DNS2 v2: Breaking a Decade of Backwards Compatibility

I have maintained Net_DNS2 since 2010. It started as a cleanup of the old PEAR Net_DNS library, and for most of its life the guiding rule was simple: do not break anyone. That rule held through PHP 5.3, 5.4, 7.x, and into 8.x. With v2.0 I broke it deliberately, and I think it was overdue.

What PEAR-Era Naming Costs You

Net_DNS2 predates PSR-4, namespaces, and any modern autoloading convention. The class naming reflected that:

$r = new Net_DNS2_Resolver(['nameservers' => ['8.8.8.8']]);
$result = $r->query('example.com', 'MX');

Underscores as a pseudo-namespace worked, and it left the library carrying an autoloader that mapped underscores to directory separators, class names that grew to Net_DNS2_RR_OPENPGPKEY, and no way to use any of the type machinery PHP had spent a decade adding. Every resource record type was a loosely typed bag of public properties. Every enumerated value was an integer or a bare string, validated by convention.

The practical cost was in the bug reports. A meaningful share were people passing the wrong type into something, getting no error, and finding out later when the wire format came out malformed. The library could not tell them, because it had no way to say what it expected.

v2.0 Requires PHP 8.1

The v2.0 rewrite moved to PSR-4 and real namespaces, with the parallel rename you would expect:

$r = new \NetDNS2\Resolver(['nameservers' => ['8.8.8.8']]);
$result = $r->query('example.com', 'MX');

The floor is PHP 8.1, which is what enums require. That version choice is the entire reason the break was worth making. DNS is a protocol built almost entirely out of small enumerated sets: record types, classes, opcodes, response codes, DNSSEC algorithms, digest types. Modelling those as integer constants means every function that accepts one accepts any integer. Modelling them as backed enums means the type system rejects nonsense before a packet is ever assembled.

Most class, method, and property names survived the move. Someone upgrading is mostly changing Net_DNS2_Thing to \NetDNS2\Thing and raising their PHP requirement, which is a mechanical change a search and replace handles. That was the design constraint I set for myself: break the naming, keep the shape.

Deciding to Break Compatibility

The argument against was straightforward. Net_DNS2 has a long tail of users on old PHP, often embedded in something they inherited and do not want to touch. A hard PHP 8.1 floor strands all of them.

What changed my mind was noticing that I was writing new code to work around the absence of types, then writing tests to catch the bugs that the absent types would have caught, then answering issues from people who hit those bugs anyway. The library was subsidising PHP 5 compatibility with permanent complexity, and the people paying that tax were mostly not the people benefiting from it.

The compromise is that v1.x still exists and still gets fixes. That is the part I would recommend to anyone in the same position. A hard break is much easier to justify when the old branch does not disappear the same day, and tagging a final v1 release that people can pin to costs almost nothing.

What I Would Do Differently

I would have done it sooner. The signal was there for years: every feature request that involved better validation ran into the same wall, and I kept routing around it. Deprecation cycles have a cost too, and carrying a compatibility promise you have quietly stopped believing in is worse than announcing the break.

I would also have been more aggressive about the enum conversion in one pass. Doing it incrementally meant a stretch where some values were enums and some were still integers, and the mixed state was more confusing than either endpoint. If you are going to break, break cleanly and finish.

If you use the library and are still on v1, there is no urgency. If you are starting something new on PHP 8.1 or later, start on v2. The DNS lookup tool on mrdns.com runs on this code, so it gets exercised against real-world responses continuously, which has caught more parsing edge cases than my test suite ever did.

QUIC in PHP, Built on OpenSSL’s Native Stack

I have spent the last while building php-quic, a PHP extension that exposes raw QUIC transport, both client and server, with first-class access to QUIC streams. The thing I am most happy about is what is not in it: no ngtcp2, no quiche, no Rust, no FFI. It is built directly on OpenSSL 3.5’s native QUIC stack, so the only dependency is an OpenSSL you very likely already have.

Why Bother

QUIC has been the transport under HTTP/3 for a few years now, and it is creeping into other protocols too: DNS-over-QUIC, SMB, and a growing list of custom things people are building when they want streams, TLS, and congestion control without standing up TCP plus a separate TLS layer. PHP had no real way to speak it. If you wanted QUIC you reached for a C library with its own TLS stack bolted on, wrapped it in FFI or a custom extension, and inherited a second copy of all the certificate and crypto logic you already trust OpenSSL to handle.

The reason this is suddenly worth doing is OpenSSL 3.5. It ships a native QUIC implementation, client and server, that reuses the same TLS engine everything else in your stack already uses. That removes the entire argument for pulling in a Rust or C QUIC library just to get a handshake. So php-quic is thin on purpose: it is a binding to a transport that is already there, not a reimplementation of QUIC.

What It Actually Does

It gives you the transport and gets out of the way. You get connections, you get streams, you get the event-loop primitive to multiplex them. What you do not get is protocol framing. HTTP/3, DNS-over-QUIC, or whatever you are building on top, that is yours to write. I made that split deliberately. The transport is the hard, fiddly, easy-to-get-wrong part, and the framing is the part that differs for every protocol.

The API is small. Three classes and a poll function:

  • Quic\Connection: a client connection, with openStream(), acceptStream(), close(), and certificate and crypto inspection.
  • Quic\Listener: the server side, bind a host and port and accept() connections.
  • Quic\Stream: the actual data path, with write(), read(), end(), and reset(). Bidirectional or unidirectional.
  • Quic\poll(): the event-loop primitive for non-blocking, multiplexed I/O across many streams.

A client connection is about as short as you would hope:

$conn = new Quic\Connection('cloudflare-quic.com', 443, ['alpn' => 'h3']);
$control = $conn->openStream(false);
$control->write("\x00\x04\x00");

That opens a QUIC connection, negotiates the h3 ALPN, and opens a unidirectional control stream. Everything past that first byte is HTTP/3 framing that you write yourself.

A Concrete Example: DNS-over-QUIC

DNS-over-QUIC is a clean fit for QUIC: one query per bidirectional stream, a two-octet length prefix, done. With php-quic the transport part collapses to a few lines:

$conn = new Quic\Connection('dns.adguard-dns.com', 853, ['alpn' => 'doq']);
$stream = $conn->openStream();
$stream->write(pack('n', strlen($query)) . $query, true);

The true on the write sends the FIN with the data, which is exactly what DoQ wants: one query, stream half-closed, read the response back. That is the whole transport. If you want to see DNS-over-QUIC running against a real resolver without any of this, mrdns.com has the diagnostic tooling.

Non-Blocking by Default

QUIC multiplexes many streams over one connection, so a blocking read-one-thing-at-a-time model throws away the entire point. The poll() primitive lets you wait on a set of streams and act on whichever is ready:

Quic\poll([[$stream, Quic\POLL_READ | Quic\POLL_ERROR]], 1.0);
$chunk = $stream->read(65535);

That is enough to drive a real event loop, and on the server side a Quic\Listener can fan out across processes with SO_REUSEPORT so you scale horizontally without a load balancer in front doing connection-aware routing.

Requirements and Install

You need PHP 8.4 or newer (8.5 on Windows) and OpenSSL 3.5.0 or newer built with the native QUIC stack. The OpenSSL version is the real gate here, since 3.5 is recent. The easy path is PIE:

pie install mikepultz/php-quic

Or build it the usual way and add extension=quic to your php.ini:

phpize
./configure --with-quic
make && make test
sudo make install

What It Does Not Do Yet

I would rather be honest about the edges than oversell this. A few QUIC features are not in the first cut: no unreliable datagrams (RFC 9221), no 0-RTT early data, and no connection migration. There is also a small per-stream memory retention, on the order of 300 bytes, that matters only on very long-lived connections opening enormous numbers of streams. None of these block the protocols I built it for, but if your use case depends on datagrams or 0-RTT, know that going in.

Try It

The code is on GitHub under BSD-3-Clause. If you have wanted to speak QUIC from PHP, whether that is HTTP/3, DoQ, or something of your own, I would genuinely like to hear what you build with it and where the API gets in your way. Issues and pull requests welcome.

Google Speech API – Full Duplex PHP Version

So this is a follow up to my post a while ago, talking about how to use the Google Speech Recognition API built in to Google Chrome.

Since my last post, Chrome has had some significant upgrades to this feature- specifically around the length of audio you can pass to the API. The old version would only let you pass very short clips (only a few seconds), but the new API is a full-duplex streaming API. What this means, is that it actually uses two HTTP connections- one POST request to upload the content as a “live” chunked stream, and a second GET request to access the results, which makes much more sense for longer audio samples, or for streaming audio.

I created a simple PHP class to access this API; while this likely won’t make sense for anybody that wants to do a real-time stream, it should satisfy most cases where people just want to send “longer” audio clips.

Before you can use this PHP class, you must get a developer API key from Google. The class does not include one, and I cannot give you one- they’re free, and easy to get just go to the Google APIs site, and sign up for one.

Then download the class below, and start with a simple example:

<? 
require 'google_speech.php';

$s = new cgoogle_speech('put your API key here'); 

$output = $s->process('@test.flac', 'en-US', 8000);      

print_r($output);
?>

Audio can be passed as a filename (by prefixing the ‘@’ sign in front of the file name), or by passing in raw FLAC content. The second argument is an IETF language tag. I’ve only been able to test with both English and French, but I assume others work. It defaults to ‘en-US’. The third argument is sample rate, it defaults to 8000.

** Your sample rate must match your file- if it doesn’t, you’ll either get nothing returned, or you’ll get a really bad transcription. **

The output will return as an array, and should look something like this:

Array
(
    [0] => Array
        (
            [alternative] => Array
                (
                    [0] => Array
                        (
                            [transcript] => my CPU is a neural net processor a learning computer
                            [confidence] => 0.74177068
                        )
                    [1] => Array
                        (
                            [transcript] => my CPU is the neuron that process of learning
                        )
                    [2] => Array
                        (
                            [transcript] => my CPU is the neural net processor a learning
                        )
                    [3] => Array
                        (
                            [transcript] => my CPU is the neuron that process a balloon
                        )
                    [4] => Array
                        (
                            [transcript] => my CPU is the neural net processor a living
                        )
                )
            [final] => 1
        )
)

Get the PHP class here: http://mikepultz.com/uploads/google_speech.php.zip

Mining Twitter API v1.1 Streams from PHP – with OAuth

This is a quick update to my post about a year ago, with details on how to mine Twitter streams in real-time using PHP. This new code includes updates for the v1.1 API, including authentication using OAuth.

The first thing you need to do is sign in to the Twitter developer portal with your Twitter account here: https://dev.twitter.com/user/login 

Once you’ve logged in, click on your profile icon in the top right hand corner, select
“My applications”, and create a new application if you don’t already have one.

Select the option to create the access token as well, as the requests need to be signed by a Twitter account.

The Code

ctwitter_stream.php

class ctwitter_stream
{
    private $m_oauth_consumer_key;
    private $m_oauth_consumer_secret;
    private $m_oauth_token;
    private $m_oauth_token_secret;

    private $m_oauth_nonce;
    private $m_oauth_signature;
    private $m_oauth_signature_method = 'HMAC-SHA1';
    private $m_oauth_timestamp;
    private $m_oauth_version = '1.0';

    public function __construct()
    {
        //
        // set a time limit to unlimited
        //
        set_time_limit(0);
    }

    //
    // set the login details
    //
    public function login($_consumer_key, $_consumer_secret, $_token, $_token_secret)
    {
        $this->m_oauth_consumer_key     = $_consumer_key;
        $this->m_oauth_consumer_secret  = $_consumer_secret;
        $this->m_oauth_token            = $_token;
        $this->m_oauth_token_secret     = $_token_secret;

        //
        // generate a nonce; we're just using a random md5() hash here.
        //
        $this->m_oauth_nonce = md5(mt_rand());

        return true;
    }

    //
    // process a tweet object from the stream
    //
    private function process_tweet(array $_data)
    {
        print_r($_data);

        return true;
    }

    //
    // the main stream manager
    //
    public function start(array $_keywords)
    {
        while(1)
        {
            $fp = fsockopen("ssl://stream.twitter.com", 443, $errno, $errstr, 30);
            if (!$fp)
            {
                echo "ERROR: Twitter Stream Error: failed to open socket";
            } else
            {
                //
                // build the data and store it so we can get a length
                //
                $data = 'track=' . rawurlencode(implode($_keywords, ','));

                //
                // store the current timestamp
                //
                $this->m_oauth_timestamp = time();

                //
                // generate the base string based on all the data
                //
                $base_string = 'POST&' . 
                    rawurlencode('https://stream.twitter.com/1.1/statuses/filter.json') . '&' .
                    rawurlencode('oauth_consumer_key=' . $this->m_oauth_consumer_key . '&' .
                        'oauth_nonce=' . $this->m_oauth_nonce . '&' .
                        'oauth_signature_method=' . $this->m_oauth_signature_method . '&' . 
                        'oauth_timestamp=' . $this->m_oauth_timestamp . '&' .
                        'oauth_token=' . $this->m_oauth_token . '&' .
                        'oauth_version=' . $this->m_oauth_version . '&' .
                        $data);

                //
                // generate the secret key to use to hash
                //
                $secret = rawurlencode($this->m_oauth_consumer_secret) . '&' . 
                    rawurlencode($this->m_oauth_token_secret);

                //
                // generate the signature using HMAC-SHA1
                //
                // hash_hmac() requires PHP >= 5.1.2 or PECL hash >= 1.1
                //
                $raw_hash = hash_hmac('sha1', $base_string, $secret, true);

                //
                // base64 then urlencode the raw hash
                //
                $this->m_oauth_signature = rawurlencode(base64_encode($raw_hash));

                //
                // build the OAuth Authorization header
                //
                $oauth = 'OAuth oauth_consumer_key="' . $this->m_oauth_consumer_key . '", ' .
                        'oauth_nonce="' . $this->m_oauth_nonce . '", ' .
                        'oauth_signature="' . $this->m_oauth_signature . '", ' .
                        'oauth_signature_method="' . $this->m_oauth_signature_method . '", ' .
                        'oauth_timestamp="' . $this->m_oauth_timestamp . '", ' .
                        'oauth_token="' . $this->m_oauth_token . '", ' .
                        'oauth_version="' . $this->m_oauth_version . '"';

                //
                // build the request
                //
                $request  = "POST /1.1/statuses/filter.json HTTP/1.1\r\n";
                $request .= "Host: stream.twitter.com\r\n";
                $request .= "Authorization: " . $oauth . "\r\n";
                $request .= "Content-Length: " . strlen($data) . "\r\n";
                $request .= "Content-Type: application/x-www-form-urlencoded\r\n\r\n";
                $request .= $data;

                //
                // write the request
                //
                fwrite($fp, $request);

                //
                // set it to non-blocking
                //
                stream_set_blocking($fp, 0);

                while(!feof($fp))
                {
                    $read   = array($fp);
                    $write  = null;
                    $except = null;

                    //
                    // select, waiting up to 10 minutes for a tweet; if we don't get one, then
                    // then reconnect, because it's possible something went wrong.
                    //
                    $res = stream_select($read, $write, $except, 600, 0);
                    if ( ($res == false) || ($res == 0) )
                    {
                        break;
                    }

                    //
                    // read the JSON object from the socket
                    //
                    $json = fgets($fp);

                    //
                    // look for a HTTP response code
                    //
                    if (strncmp($json, 'HTTP/1.1', 8) == 0)
                    {
                        $json = trim($json);
                        if ($json != 'HTTP/1.1 200 OK')
                        {
                            echo 'ERROR: ' . $json . "\n";
                            return false;
                        }
                    }

                    //
                    // if there is some data, then process it
                    //
                    if ( ($json !== false) && (strlen($json) > 0) )
                    {
                        //
                        // decode the socket to a PHP array
                        //
                        $data = json_decode($json, true);
                        if ($data)
                        {
                            //
                            // process it
                            //
                            $this->process_tweet($data);
                        }
                    }
                }
            }

            fclose($fp);
            sleep(10);
        }

        return;
    }
};

The “process_tweet()” method will be called for each matching tweet- just modify that method to process the tweet however you want (load it into a database, print it to screen, email it, etc). The keyword matching isn’t perfect- if you search for a string of words, it won’t necessarily match the words in that exact order, but you can check that yourself from the process_tweet() method.

Then create a simple PHP application to run the collector:

require 'ctwitter_stream.php';

$t = new ctwitter_stream();

$t->login('consumer_key', 'consumer secret', 'access token', 'access secret');

$t->start(array('facebook', 'fbook', 'fb'));

You’ll need to provide the Consumer Key, Consumer Secret, Access Token, and the Access Secret, all of which are available from the Details section of your Application.

This new class uses the PHP hash_hmac() function for OAuth, which is available only in PHP 5.2.1 and up, and in the PECL hash extension 1.1 and up.

You can also Download the file here: http://mikepultz.com/uploads/ctwitter_stream.php.zip