Fanning Out 140 DNS Queries per Host Without Melting the Server

A blacklist check is not one DNS query. Checking a single IP against our source list means somewhere north of 140 DNS lookups, and the checker runs continuously across a large number of hosts. The naive implementation of that is a loop calling getaddrinfo(), and it is catastrophically slow for a reason that has nothing to do with CPU.

DNS Is Almost Entirely Waiting

A DNSBL lookup is a small UDP packet out and a small UDP packet back. The work is negligible. What dominates is latency, and the distribution is ugly: most zones answer in tens of milliseconds, a few take hundreds, and some fraction never answer at all and burn your full timeout. Serialized, 140 lookups at an average of 60ms is over eight seconds for one host, and the tail is far worse than the average.

The instinct is a thread per query. That works until it doesn’t. Threads are expensive relative to the work being done, and a thread blocked on a 5 second DNS timeout is a thread doing nothing while holding a stack. Scale that across concurrent hosts and you are context-switching more than resolving.

One Socket, Many Queries in Flight

DNS over UDP is already designed for multiplexing. Every query carries a 16-bit ID, and the response echoes it back. That gives you everything you need to run hundreds of queries over a single socket and demultiplex the answers as they arrive:

//
// send all queries first, then collect
//
for (auto const &source : this->m_sources)
{
    uint16_t id = this->_next_query_id();

    this->m_pending[id] = { source.id, now_ms() + source.timeout_ms };
    this->_send_query(id, source, _host);
}

Then a single epoll_wait() loop reads whatever comes back, looks up the ID, and retires that entry. The whole batch costs one socket and one thread, and wall clock becomes the slowest response rather than the sum of all of them.

Two details make this correct rather than merely fast. The query ID has to be random, not sequential, because a predictable ID plus a known source port is the cache-poisoning setup from 2008. And you have to verify that the response actually matches the question you asked, since the socket is unconnected and will happily hand you a packet from anyone.

Timeouts Are the Hard Part

Nothing tells you a UDP query failed. The absence of a response is the signal, which means every in-flight query needs a deadline and something has to sweep for expired ones. Checking all 140 pending entries on every loop iteration is fine at this scale. At larger scale, keep them in a structure ordered by deadline so the sweep only touches what has actually expired.

int64_t now = now_ms();

for (auto it = this->m_pending.begin(); it != this->m_pending.end(); )
{
    if (it->second.deadline <= now)
    {
        this->_record_timeout(it->second.source_id);
        it = this->m_pending.erase(it);
    } else
    {
        ++it;
    }
}

The timeout value itself is a judgment call with real consequences. Too short and you record false negatives for slow-but-working zones, which quietly degrades your results. Too long and one dead source sets the floor for every check. Per-source timeouts, tuned from that source’s observed behavior, beat one global number.

The Failure Modes That Only Show Up in Production

Source port exhaustion is the first one. If you open a socket per query, you will run out of ephemeral ports under load, and the symptom is intermittent bind() failures that look like nothing else. A small pool of sockets with many queries multiplexed over each avoids it entirely.

Rate limiting is the second. Public DNSBL operators will throttle or block you, and they are right to. Anything running at volume belongs on a paid data feed or a local mirror, and the checker needs per-source concurrency caps so one aggressive batch cannot trip a limit that then poisons results for every subsequent check.

The third one took me longest to appreciate. A source that returns NXDOMAIN and a source that never answers look similar in a naive implementation, and they mean completely different things. The first is a confirmed “not listed”. The second is “no data”, and recording it as “not listed” turns an outage at one provider into a silent gap in your results. Track them as distinct outcomes from the start, because retrofitting that distinction after the fact means every historical row is ambiguous.