Tag Archives: AI

Where AI Compute Goes When the Router Does the Work

My last post ended on a small, quiet machine on the shelf beside the Wi-Fi router. Six months earlier, the IEEE 802.11 working group, which writes the Wi-Fi standard, had voted to start letting the router be that machine, or at least find it for you.

What the Standard Actually Covers

Qualcomm brought the proposal, now called AI Offload, in March, co-signed by Cisco, Meta, MediaTek, Huawei, Sony, Nokia and others. A device that can’t run an AI model (the proposal’s example is “a non-AI Wi-Fi home security camera”) sends the job to a router that can and gets the answer back. It can even tell the router where to download the model, so a camera maker can ship its own AI without an AI chip in the camera.

Routers would announce the service the way they announce their network name, so a device can see what AI is on offer (text, voice, images, video) before it connects.

By July the scope had grown. The draft charter covers AI work done by the router, by another device on your Wi-Fi, or by a device plugged into your network behind the router: the box on the shelf. MediaTek has proposed how that would work: AI boxes check in with the router, and the router passes your request to whichever one can take it.

The standard covers finding the service, connecting to it, and securing the connection. The AI work itself is out of scope; Cisco’s review of the early proposals calls the model “a black box” as far as Wi-Fi is concerned. That’s the right call. A Wi-Fi standard that tried to pin down AI models would be obsolete before it shipped.

When What happened
January 2026 Broadcom sends router makers early samples of the BCM4918, a Wi-Fi 8 router chip with built-in AI hardware
March 2026 Qualcomm announces the Dragonwing NPro A8 Elite, a router chip with its own AI processor; the Wi-Fi working group votes to start on AI Offload
May 2026 The AI Offload group meets for the first time, in Antwerp
September 2026 The working group approves the charter; final sign-off expected in November
July 2029 Target for the first formal vote on a draft standard
July 2030 Target for the finished standard

Why Your Camera Shouldn’t Need Its Own Brain

Today a smart home device gets its AI from a chip inside it or from its maker’s cloud.

Eufy’s $299.99 E40 lock does the first: it stores up to 50 faces, and “all facial data is stored locally.” That works, but every smart device then needs its own AI chip, built from scratch by every company.

Ring does the second. Its Familiar Faces feature tags people at your door, and Amazon says it “processes and stores biometrics collected by Ring cameras on its own servers.” It isn’t offered in Illinois, Texas, or Portland, Oregon, all of which have strict biometric privacy laws.

The cloud can also disappear. When Belkin shut down Wemo’s cloud on January 31, 2026, the app, remote access, and the Alexa and Google integrations went with it. Only the Wemo devices already set up in Apple’s HomeKit kept working.

The router is a third option. A $30 camera with no AI hardware sends its frames to the router, the router runs the model, and the faces stay in the house. If the camera company goes under, the model’s still on your router. The models are small, too: Frigate, a free, open-source recorder for home security cameras, added face and license plate recognition last year, and its small face model “runs locally on the CPU” of an ordinary computer.

The obvious example is the front door: the doorbell camera sees you coming, the router matches your face, and the lock opens before you find your keys. I wouldn’t do it on the face alone. A camera can’t always tell a face from a good photo of one, and a cheap camera won’t have the 3D sensor your phone uses for face unlock. But the router knows your phone just joined the network. Face plus phone beats either one alone (e.g. MFA for your front door).

Two-factor front door: the doorbell camera sends a face and your phone joins the Wi-Fi; the router checks that the face matches someone who lives here and that their phone is on the network, and only then unlocks the door. A photo held up to the camera passes the face check but fails the phone check, so the door stays locked.

Once a cheap device can borrow the router’s model, the list of things worth building gets long. A few:

Devices What the router’s model does
Driveway camera, garage door or gate Reads the plate and opens for the family car. Nobody needs a giant cloud AI to read a license plate.
Porch camera, phone “The FedEx driver left a box” instead of “motion detected,” and another alert if someone else walks off with it.
Cheap smart speaker The speaker only listens for its wake word, like “Alexa”; the router does the rest.
Smoke and CO alarms, lights, heating and cooling Any microphone in the house hears the alarm; the router turns on the lights and shuts off the furnace fan so it stops pushing smoke through the ducts.
Wi-Fi motion sensing, family alerts Spots a fall in a parent’s house from the way a body disturbs the Wi-Fi signal, with no camera in the bathroom. Wi-Fi sensing got its own standard, 802.11bf, in 2025.

Most of these exist today, each locked to one vendor’s app and cloud. The standard would let a camera from one company use a router from another, the way any Wi-Fi phone works with any Wi-Fi router.

Your Laptop Uses the Same Box

Cameras make the easy case, but most of my AI questions come from my laptop. Lenovo’s scope proposal lists laptops as offload clients too, to save battery and run models too big for them. The announcement that lets a camera find the router lets a laptop find the shelf box, so your AI apps can use it by default and go to the cloud only when the policy allows. The software is mostly there: Ollama, a free tool for running models at home, accepts the same requests as OpenAI’s service, so many apps can switch to it by changing one address.

The bigger change is what that box can see. MCP, an open standard for AI apps, lets you plug your calendar or files into a model so it can answer questions about them. At home, you plug in the house: every camera and sensor, what they’ve recorded, and what they can do. Home Assistant, the free home automation platform, already offers its devices to Claude and ChatGPT this way, and the charter’s supporting document describes a router that keeps a directory of the AI agents on the network.

Then your laptop can ask about the house, using data that never left it:

  • “Who came to the door while I was out?”
  • “Was the garage door open overnight this week?”
  • “Are we using more water this month than last?”

A cloud model could answer those too, if you sent it your doorbell footage and your water readings.

Sending Work to the Big Cloud Models Isn’t in the Standard

Some work still wants a frontier model, one of the big cloud AIs like ChatGPT or Claude: summarizing a contract, or anything that needs real reasoning. The standard doesn’t cover sending it there, but it did come up.

Lenovo’s proposal listed handing work to a cloud provider as one possible design, and Huawei argued that the biggest models “have to be deployed on the cloud.” The draft charter went the other way. It only counts a device “capable of fulfilling an AI offload request without further assistance from another device,” which, as I read it, rules out a router that just forwards your request to a cloud service. Cisco’s review said where the AI runs is something ordinary networking already handles.

That’s right for a Wi-Fi standard, and it doesn’t get in the way. The box behind the router can answer what it can and send the rest to the cloud. That’s the idea from my first post, sending each request to the cheapest model that can handle it, moved inside the house.

Home Assistant already works this way by default: its built-in assistant handles commands first, and “only questions or commands it can’t understand will be sent to the AI you’ve set up.” Apple Intelligence runs many models on the device, sends harder requests to “larger, server-based models” on Apple’s servers, and for ChatGPT, “users are asked before any questions are sent.”

Tier Where it runs Typical work Leaves the house?
Device The camera, speaker, or glasses Wake word, motion trigger No
Router AI chip in the router Face match, plate read, voice commands No
Shelf box Plugged into your home network Your laptop’s questions, the family’s mail and photos No
Cloud Someone else’s data center Hard reasoning, long research Yes, when the house allows it

The last column is why I want this. Today each device maker decides what goes to its cloud. If requests leave through the router, the house decides, in one policy. Nothing defines that policy yet; here’s roughly what I’d want mine to say:

One policy for the whole house: faces and license plates go to the router and never leave the house; mail, calendar and photos go to the shelf box and stay home; everything else goes to the big AI models in the cloud, but only after asking first.

The Four Years Before 2030

The chips are in router makers’ hands now. The standard lands in 2030 if the schedule holds, and Wi-Fi schedules tend to slip. Until then, every router maker will ship its own version, every camera maker will pair with whoever pays them, and you’ll get a doorbell that only talks to one brand of router. Smart home hubs have worked that way for a decade.

Memory is still the hard part. An AI chip that reads license plates is cheap; a router with enough memory to run a large AI model costs what I complained about last time. That’s why the clause I care most about is the box plugged into your home network. The router only has to know where the big model is, and the box can sit on the shelf and get cheaper on its own schedule.

If the standard ships close to today’s charter, every device in the house will find the box from my last post the same way it finds the Wi-Fi.

Where AI Compute Goes When It Comes Home

In my last post I argued that once model progress levels off, inference moves toward the people using it. The closest person using it is sitting in your living room. So I went and counted the hardware built to put a model there: 21 desktop AI boxes announced or shipped in the last twelve months, from 16 companies. Most launched somewhere between $2,000 and $4,000.

Twenty-One Boxes in a Year

Nvidia announced Project DIGITS at CES in January 2025: 128GB of unified memory for “$3,000,” due in May. Apple shipped first. Two months later the Mac Studio could be configured with up to 512GB, which Apple called “the most unified memory ever in a personal computer,” and it pitched the machine as able to run models “with over 600 billion parameters entirely in memory.” DIGITS finally went on sale in October as the DGX Spark, at $3,999.

Then everyone piled in. Seven PC makers sell their own version of the Spark’s GB10 board. AMD’s Ryzen AI Max+ 395, the chip everyone calls Strix Halo, ended up in mini PCs from Framework and a long list of smaller brands, and this summer AMD started selling its own box through Micro Center. On August 25 Apple announced the M5 Mac Studio, shipping September 22.

Platform Sold by Max memory Memory bandwidth Price
Nvidia GB10 Nvidia, Acer, ASUS, Dell, Gigabyte, HP, Lenovo, MSI 128GB 273GB/s DGX Spark: $3,999 at launch, $4,699 since February
AMD Ryzen AI Max+ 395 AMD, Framework, many mini PC brands 128GB 256GB/s AMD Ryzen AI Halo: $3,999.99
Apple M5 Max Apple (Mac Studio) 128GB 614GB/s From $2,499
Apple M5 Ultra Apple (Mac Studio) 512GB 1.2TB/s From $5,499; 512GB config due late October

Then Memory Got Expensive

What makes these boxes useful is a big pool of fast memory, and memory is the part that blew up. TrendForce says conventional DRAM contract prices rose roughly 93% to 98% in the first quarter of 2026 alone, with another 58% to 63% forecast for the second. It puts the blame on AI servers soaking up general-purpose memory. The same build-out I wrote about last time is buying the same chips.

Device Launch price Price now
Raspberry Pi 5, 16GB $120 (January 2025) $305
Framework Desktop, 128GB $1,999 (February 2025) $3,449, out of stock
Mac Studio, base $1,999 (March 2025) $2,499 (June 2026)
Mac Studio, M3 Ultra $3,999 (March 2025) $5,299 (June 2026)
Nvidia DGX Spark $3,999 (October 2025) $4,699 (February 2026)

Tim Cook called it a “hundred-year flood” when Apple raised Mac prices in June. Raspberry Pi says the LPDDR4 on its boards went up seven-fold in a year.

The Box Next to the Router

Here’s the world I think these boxes point to. Every house has a small, quiet machine on the shelf beside the Wi-Fi router. It holds the family’s mail, photos, documents, and calendar, and it runs a model good enough to answer questions about all of it. Nothing leaves the house. There’s no per-token bill, and no status page to refresh when a provider has a bad afternoon.

The worry is already there. In Pew’s February 2026 survey, roughly seven in ten Americans said AI will make their personal information less secure. Apple is selling the new Mac Studio on exactly that: its launch copy says you can “run massive models entirely on device with complete privacy.” The biggest home assistant went the other way. In March 2025 Amazon removed the “Do Not Send Voice Recordings” option from several Echo devices, because its generative Alexa features “rely on the processing power of Amazon’s secure cloud.”

Power is a practical problem. A Netgear Orbi router idles at 7.4W. ServeTheHome measured a DGX Spark idling at 40 to 45W and drawing 60 to 90W during LLM inference. That’s fine on a developer’s desk. It’s a harder sell for something that runs all day in a hallway closet.

Price is the bigger one. Nobody puts a $4,699 box next to their router. The home version has to cost about what a good router costs, so it can’t depend on a big GPU or a giant pool of premium unified memory. It has to run on ordinary hardware. That sounds out of reach today. I don’t think it is.

Memory Bandwidth Sets the Speed Limit

Generating text on a local model is mostly a memory problem. For every token, the machine reads the model’s active weights out of memory, so the ceiling on speed is roughly how fast it can move bytes, divided by how many bytes each token needs.

tokens/sec ceiling  ~  memory bandwidth / bytes read per token

dense 70B, 4-bit            ~40 GB per token
  Strix Halo @ 256 GB/s     ~6 tok/s ceiling       measured: 5.0

MoE, 3B active, 4-bit       ~2 GB per token
  Strix Halo @ 256 GB/s     ~130 tok/s ceiling     measured: 72.0

Apple’s ML team showed this cleanly when it tested the M5. Memory bandwidth went from 120GB/s on the M4 to 153GB/s, a 28% bump, and token generation got 19% to 27% faster. Time to first token is compute-bound, and it improved 3.3x to 4x.

Here’s where the hardware lands.

Hardware Memory bandwidth
Desktop, dual-channel DDR5-5600 ~90GB/s on paper
AMD Ryzen AI Max+ 395 256GB/s
Nvidia GB10 273GB/s
Apple M4 Max 410 to 546GB/s
Apple M5 Max 614GB/s
Apple M3 Ultra 819GB/s
Apple M5 Ultra 1.2TB/s

An ordinary desktop gets you about a third of a Spark, and the top Mac Studio is more than four times past the Spark. On bandwidth alone, cheap hardware loses badly.

Mixture of Experts Changed the Math

A mixture-of-experts (MoE) model splits its weights into many small expert networks and only runs a few of them per token. Qwen3.6-35B-A3B has 35B parameters in total and activates 3B. Total parameters decide how much memory you need. Active parameters decide how fast it runs. That split is the whole case for cheap hardware, because ordinary DDR5 is slow but you can put a lot of it in an ordinary machine.

Hardware Dense model Speed MoE model Speed
Nvidia DGX Spark Llama 3.1 70B, FP8 2.7 tok/s gpt-oss-120b, MXFP4 58.7 tok/s
AMD Strix Halo Llama 3 70B fine-tune, Q4_K_M 5.0 tok/s Qwen3-30B-A3B, Q4 72.0 tok/s
Laptop, dual-channel DDR5-5600, CPU only Qwen2.5-Coder 32B 3.5 tok/s Qwen3-Coder-Next 80B-A3B 7.7 tok/s
Raspberry Pi 5, 16GB Qwen3-30B-A3B, 2.7 bits per weight 8.0 tok/s

These numbers come from different people using different tools, so don’t read them too precisely. The gaps are too big to be noise, though: the same Spark runs a 117B MoE model more than 20 times faster than a dense 70B.

The laptop row is the one I care about. It’s a CPU with ordinary RAM, and an 80B MoE model runs twice as fast on it as a dense 32B. The person who posted those numbers says the MoE result is still 3 to 4 times slower than the bandwidth math predicts, so the software has room left. Then there’s the Pi: a $305 board running a 30B model at 8 tokens a second, at a quantization that keeps about 94% of full-precision quality.

The software is closing that gap quickly. llama.cpp added --cpu-moe in August 2025, which keeps the expert weights in system RAM and puts the rest on whatever GPU you have. Running Qwen3.6-35B-A3B that way is one line:

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M -ngl 99 --cpu-moe

One user runs a community fine-tune of that model this way on a Ryzen 7 5800X with a 6GB GTX 1660 Super, and gets almost 16 tokens a second. That’s a gaming PC from 2020.

Multi-token prediction landed in llama.cpp in May 2026. On an RTX 3090, Qwen3.6-27B went from 23 to 42 tokens a second from a software update. Apple’s M4-to-M5 jump bought 19% to 27%.

The models keep shrinking for the same capability, too. The Densing Law paper estimates the capability density of LLMs doubles roughly every three months. Epoch AI found a single RTX 5090 runs models that matched the frontier 6 to 12 months earlier, and it notes its method is biased against sparse models, so MoE likely shortens that lag.

What Still Doesn’t Work on Cheap Hardware

Reading the prompt. Generation is bound by bandwidth, but prompt processing is bound by compute, and CPUs are slow at it. On a 48-core EPYC with 12 channels of DDR5, CPU only, gpt-oss-120b processes prompts at about 109 tokens a second. A 32K-token prompt means at least five minutes before the first word comes back. The Spark’s GPU does the same model at about 2,400. For a chat, that’s fine. For “summarize my whole inbox,” a CPU box has to do the reading ahead of time, overnight, while nobody’s waiting. For something that sits next to the router all day, that’s a reasonable design.

New architectures. On that same EPYC, Qwen3-Next-80B-A3B, which uses a newer hybrid attention design, ran at 11.8 tokens a second. Qwen3-30B-A3B, with about the same active size, ran at 63.1. The software catches up, but the cheapest hardware tends to get the fast path last.

Quality. The Pi result keeps 94% of full-precision quality, which also means it gives up 6%. For a house assistant answering questions about the calendar, I’d take that trade. For hard reasoning, I wouldn’t.

Memory prices. The cheap-hardware argument depends on plenty of ordinary RAM, and the LPDDR4 on that Pi is up seven-fold in a year. The Pi 5 16GB launched at $120 in January 2025. It’s $305 now.

My bet is that the home AI box ends up being a cheap machine with a lot of ordinary memory, running a MoE model and doing its heavy reading while the house sleeps. The $4,000 developer boxes are proving the demand and paying for the software work that gets there. The silicon exists. The models are getting there on a three-month doubling. What’s holding it back in 2026 is the price of DRAM, and that’s a supply problem.

When supply catches up, I expect the box next to the router to cost about what the router did.

Where AI Compute Goes When the Models Stop Getting Better

Almost every AI infrastructure bet I see rests on two assumptions: that the best models keep getting meaningfully better, and that you keep renting them from three or four companies. Both are weaker than they look, and I think they break sooner than the roadmaps imply. Here’s the chain of reasoning, in the order I expect it to play out.

The Frontier Is Getting Crowded From Below

The distance between the best closed model and the best open one is now measured in months. Epoch AI puts the lag at roughly four months, about one point release, and Stanford’s 2026 AI Index shows four of the top ten public Arena models are now open weight, clustered tight on Elo.

The Frontier Is Getting Crowded From Below

The Frontier Is Getting Crowded From Below

Alibaba’s Qwen3.8-Max matches the closed frontier on general knowledge in its own benchmarks, and its open-weight sibling fits on a single GPU.

The frontier is still closed. But “a few percent behind and you can download the weights” is a different market than “only three companies can do this,” and everything below follows from that.

Nobody Needs a Frontier Model to Read a License Plate

Most production AI is narrow: translate a field, pull a total off an invoice, read a license plate, classify a ticket. Aiming a frontier model at those is like renting a rack to run cron. The cheaper purpose-tuned model that clears the bar at a fraction of the cost is the right call, so architectures are going multi-model: route each request to the cheapest model that passes, keep the expensive one for the genuinely hard reasoning.

The clearest signal of where this is headed is that Stripe just agreed to acquire OpenRouter (announced August 19, terms undisclosed, reported north of $7B by Bloomberg and TechCrunch), the router that sits in front of 400+ models and meters the tokens. When a payments company pays billions for the thing that routes and bills model calls, tokens have become a commodity you buy by the unit.

Commodities sold through a broker that shops for the cheapest passing option only get cheaper. If your business model assumes today’s token prices, plan for them falling by an order of magnitude.

The Plateau Comes Sooner Than the Roadmaps Say

Model progress will plateau, and given how fast the curve has moved, it will be sooner than you think. When the frontier stops jumping, the game stops being about the “smartest model” and becomes “same quality for the least money, power, and space,” which is a hardware question.

Today the answer is Nvidia H100 and H200 GPUs, and the challenge is two things: programmability (CUDA, a chip that runs whatever you invent next quarter) and networking (NVLink and InfiniBand lash thousands of GPUs into one fabric). But a frozen architecture is exactly what you burn into silicon, and plenty of companies are already betting on that.

Companies Already Building Fixed-Function AI Silicon

Companies Already Building Fixed-Function AI Silicon

Etched is the purest bet: Sohu hard-codes the transformer into silicon and runs nothing else. Its throughput claims are marketing until someone benchmarks them independently, but the thesis holds. If the transformer is the architecture for the next decade, a chip that does only transformers wins on cost per token by a margin a flexible chip can’t answer.

We’ve seen this arc in crypto mining: CPU, then GPU, then ASIC, each stage wiping out the last. Litecoin’s Scrypt was deliberately designed to be “ASIC-resistant” to keep mining on GPUs. It didn’t work; Bitmain shipped the Antminer L3, a Scrypt ASIC, and the holdouts were finished.

AI is harder than a hash, so this runs slower and messier, and Nvidia’s networking outlasts its CUDA one. But the people betting GPUs stay central forever are standing where the Scrypt holdouts stood. It’s a question of when, and the when is closer than the capex suggests.

What Happens to the Buildings

Meanwhile the industry is pouring concrete for a frontier that keeps advancing and a GPU that stays central.

What's Been Announced (proposed data centers)

What’s Been Announced (proposed data centers)

I don’t believe most of this gets built: Morgan Stanley pegs the data-center financing gap through 2028 at around $1.5T, Bain figures the industry is roughly $800B a year short on the revenue to fund it, and Satya Nadella has already said “there will be an overbuild.” Microsoft has been quietly canceling leases, which TD Cowen read as oversupply.

Now layer on the hardware shift. These campuses sit where power is cheap and land is empty, far from people, which is fine for training because training doesn’t care about latency. Inference does; it wants to sit near users.

So picture the end state: progress plateaus, the workload tilts to efficient inference on purpose-built silicon, and that inference wants a metro. What happens to a two-gigawatt training barn in rural Louisiana, built for GPUs doing a job that moved somewhere else? My honest guess is that some become the abandoned malls of this decade, stranded too far from anyone to repurpose. The ones near grid capacity, fiber, and people convert fine. The rest hold hardware that lost on cost per token, in a place nobody needs.

This is the ordinary shape of an infrastructure boom, and AI keeps working fine right through it. The first wave overbuilds general-purpose capacity chasing a moving frontier; the workload commoditizes and specializes; the hardware goes from flexible to fixed; and the winners are whoever got cost per useful token lowest, not whoever had the biggest model.

We ran this loop in mining, and in the fiber glut before it. Worth positioning for the back half now, while everyone else builds for the front.