Tag Archives: GPU

Where AI Compute Goes When It Comes Home

In my last post I argued that once model progress levels off, inference moves toward the people using it. The closest person using it is sitting in your living room. So I went and counted the hardware built to put a model there: 21 desktop AI boxes announced or shipped in the last twelve months, from 16 companies. Most launched somewhere between $2,000 and $4,000.

Twenty-One Boxes in a Year

Nvidia announced Project DIGITS at CES in January 2025: 128GB of unified memory for “$3,000,” due in May. Apple shipped first. Two months later the Mac Studio could be configured with up to 512GB, which Apple called “the most unified memory ever in a personal computer,” and it pitched the machine as able to run models “with over 600 billion parameters entirely in memory.” DIGITS finally went on sale in October as the DGX Spark, at $3,999.

Then everyone piled in. Seven PC makers sell their own version of the Spark’s GB10 board. AMD’s Ryzen AI Max+ 395, the chip everyone calls Strix Halo, ended up in mini PCs from Framework and a long list of smaller brands, and this summer AMD started selling its own box through Micro Center. On August 25 Apple announced the M5 Mac Studio, shipping September 22.

Platform Sold by Max memory Memory bandwidth Price
Nvidia GB10 Nvidia, Acer, ASUS, Dell, Gigabyte, HP, Lenovo, MSI 128GB 273GB/s DGX Spark: $3,999 at launch, $4,699 since February
AMD Ryzen AI Max+ 395 AMD, Framework, many mini PC brands 128GB 256GB/s AMD Ryzen AI Halo: $3,999.99
Apple M5 Max Apple (Mac Studio) 128GB 614GB/s From $2,499
Apple M5 Ultra Apple (Mac Studio) 512GB 1.2TB/s From $5,499; 512GB config due late October

Then Memory Got Expensive

What makes these boxes useful is a big pool of fast memory, and memory is the part that blew up. TrendForce says conventional DRAM contract prices rose roughly 93% to 98% in the first quarter of 2026 alone, with another 58% to 63% forecast for the second. It puts the blame on AI servers soaking up general-purpose memory. The same build-out I wrote about last time is buying the same chips.

Device Launch price Price now
Raspberry Pi 5, 16GB $120 (January 2025) $305
Framework Desktop, 128GB $1,999 (February 2025) $3,449, out of stock
Mac Studio, base $1,999 (March 2025) $2,499 (June 2026)
Mac Studio, M3 Ultra $3,999 (March 2025) $5,299 (June 2026)
Nvidia DGX Spark $3,999 (October 2025) $4,699 (February 2026)

Tim Cook called it a “hundred-year flood” when Apple raised Mac prices in June. Raspberry Pi says the LPDDR4 on its boards went up seven-fold in a year.

The Box Next to the Router

Here’s the world I think these boxes point to. Every house has a small, quiet machine on the shelf beside the Wi-Fi router. It holds the family’s mail, photos, documents, and calendar, and it runs a model good enough to answer questions about all of it. Nothing leaves the house. There’s no per-token bill, and no status page to refresh when a provider has a bad afternoon.

The worry is already there. In Pew’s February 2026 survey, roughly seven in ten Americans said AI will make their personal information less secure. Apple is selling the new Mac Studio on exactly that: its launch copy says you can “run massive models entirely on device with complete privacy.” The biggest home assistant went the other way. In March 2025 Amazon removed the “Do Not Send Voice Recordings” option from several Echo devices, because its generative Alexa features “rely on the processing power of Amazon’s secure cloud.”

Power is a practical problem. A Netgear Orbi router idles at 7.4W. ServeTheHome measured a DGX Spark idling at 40 to 45W and drawing 60 to 90W during LLM inference. That’s fine on a developer’s desk. It’s a harder sell for something that runs all day in a hallway closet.

Price is the bigger one. Nobody puts a $4,699 box next to their router. The home version has to cost about what a good router costs, so it can’t depend on a big GPU or a giant pool of premium unified memory. It has to run on ordinary hardware. That sounds out of reach today. I don’t think it is.

Memory Bandwidth Sets the Speed Limit

Generating text on a local model is mostly a memory problem. For every token, the machine reads the model’s active weights out of memory, so the ceiling on speed is roughly how fast it can move bytes, divided by how many bytes each token needs.

tokens/sec ceiling  ~  memory bandwidth / bytes read per token

dense 70B, 4-bit            ~40 GB per token
  Strix Halo @ 256 GB/s     ~6 tok/s ceiling       measured: 5.0

MoE, 3B active, 4-bit       ~2 GB per token
  Strix Halo @ 256 GB/s     ~130 tok/s ceiling     measured: 72.0

Apple’s ML team showed this cleanly when it tested the M5. Memory bandwidth went from 120GB/s on the M4 to 153GB/s, a 28% bump, and token generation got 19% to 27% faster. Time to first token is compute-bound, and it improved 3.3x to 4x.

Here’s where the hardware lands.

Hardware Memory bandwidth
Desktop, dual-channel DDR5-5600 ~90GB/s on paper
AMD Ryzen AI Max+ 395 256GB/s
Nvidia GB10 273GB/s
Apple M4 Max 410 to 546GB/s
Apple M5 Max 614GB/s
Apple M3 Ultra 819GB/s
Apple M5 Ultra 1.2TB/s

An ordinary desktop gets you about a third of a Spark, and the top Mac Studio is more than four times past the Spark. On bandwidth alone, cheap hardware loses badly.

Mixture of Experts Changed the Math

A mixture-of-experts (MoE) model splits its weights into many small expert networks and only runs a few of them per token. Qwen3.6-35B-A3B has 35B parameters in total and activates 3B. Total parameters decide how much memory you need. Active parameters decide how fast it runs. That split is the whole case for cheap hardware, because ordinary DDR5 is slow but you can put a lot of it in an ordinary machine.

Hardware Dense model Speed MoE model Speed
Nvidia DGX Spark Llama 3.1 70B, FP8 2.7 tok/s gpt-oss-120b, MXFP4 58.7 tok/s
AMD Strix Halo Llama 3 70B fine-tune, Q4_K_M 5.0 tok/s Qwen3-30B-A3B, Q4 72.0 tok/s
Laptop, dual-channel DDR5-5600, CPU only Qwen2.5-Coder 32B 3.5 tok/s Qwen3-Coder-Next 80B-A3B 7.7 tok/s
Raspberry Pi 5, 16GB Qwen3-30B-A3B, 2.7 bits per weight 8.0 tok/s

These numbers come from different people using different tools, so don’t read them too precisely. The gaps are too big to be noise, though: the same Spark runs a 117B MoE model more than 20 times faster than a dense 70B.

The laptop row is the one I care about. It’s a CPU with ordinary RAM, and an 80B MoE model runs twice as fast on it as a dense 32B. The person who posted those numbers says the MoE result is still 3 to 4 times slower than the bandwidth math predicts, so the software has room left. Then there’s the Pi: a $305 board running a 30B model at 8 tokens a second, at a quantization that keeps about 94% of full-precision quality.

The software is closing that gap quickly. llama.cpp added --cpu-moe in August 2025, which keeps the expert weights in system RAM and puts the rest on whatever GPU you have. Running Qwen3.6-35B-A3B that way is one line:

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M -ngl 99 --cpu-moe

One user runs a community fine-tune of that model this way on a Ryzen 7 5800X with a 6GB GTX 1660 Super, and gets almost 16 tokens a second. That’s a gaming PC from 2020.

Multi-token prediction landed in llama.cpp in May 2026. On an RTX 3090, Qwen3.6-27B went from 23 to 42 tokens a second from a software update. Apple’s M4-to-M5 jump bought 19% to 27%.

The models keep shrinking for the same capability, too. The Densing Law paper estimates the capability density of LLMs doubles roughly every three months. Epoch AI found a single RTX 5090 runs models that matched the frontier 6 to 12 months earlier, and it notes its method is biased against sparse models, so MoE likely shortens that lag.

What Still Doesn’t Work on Cheap Hardware

Reading the prompt. Generation is bound by bandwidth, but prompt processing is bound by compute, and CPUs are slow at it. On a 48-core EPYC with 12 channels of DDR5, CPU only, gpt-oss-120b processes prompts at about 109 tokens a second. A 32K-token prompt means at least five minutes before the first word comes back. The Spark’s GPU does the same model at about 2,400. For a chat, that’s fine. For “summarize my whole inbox,” a CPU box has to do the reading ahead of time, overnight, while nobody’s waiting. For something that sits next to the router all day, that’s a reasonable design.

New architectures. On that same EPYC, Qwen3-Next-80B-A3B, which uses a newer hybrid attention design, ran at 11.8 tokens a second. Qwen3-30B-A3B, with about the same active size, ran at 63.1. The software catches up, but the cheapest hardware tends to get the fast path last.

Quality. The Pi result keeps 94% of full-precision quality, which also means it gives up 6%. For a house assistant answering questions about the calendar, I’d take that trade. For hard reasoning, I wouldn’t.

Memory prices. The cheap-hardware argument depends on plenty of ordinary RAM, and the LPDDR4 on that Pi is up seven-fold in a year. The Pi 5 16GB launched at $120 in January 2025. It’s $305 now.

My bet is that the home AI box ends up being a cheap machine with a lot of ordinary memory, running a MoE model and doing its heavy reading while the house sleeps. The $4,000 developer boxes are proving the demand and paying for the software work that gets there. The silicon exists. The models are getting there on a three-month doubling. What’s holding it back in 2026 is the price of DRAM, and that’s a supply problem.

When supply catches up, I expect the box next to the router to cost about what the router did.

Where AI Compute Goes When the Models Stop Getting Better

Almost every AI infrastructure bet I see rests on two assumptions: that the best models keep getting meaningfully better, and that you keep renting them from three or four companies. Both are weaker than they look, and I think they break sooner than the roadmaps imply. Here’s the chain of reasoning, in the order I expect it to play out.

The Frontier Is Getting Crowded From Below

The distance between the best closed model and the best open one is now measured in months. Epoch AI puts the lag at roughly four months, about one point release, and Stanford’s 2026 AI Index shows four of the top ten public Arena models are now open weight, clustered tight on Elo.

The Frontier Is Getting Crowded From Below

The Frontier Is Getting Crowded From Below

Alibaba’s Qwen3.8-Max matches the closed frontier on general knowledge in its own benchmarks, and its open-weight sibling fits on a single GPU.

The frontier is still closed. But “a few percent behind and you can download the weights” is a different market than “only three companies can do this,” and everything below follows from that.

Nobody Needs a Frontier Model to Read a License Plate

Most production AI is narrow: translate a field, pull a total off an invoice, read a license plate, classify a ticket. Aiming a frontier model at those is like renting a rack to run cron. The cheaper purpose-tuned model that clears the bar at a fraction of the cost is the right call, so architectures are going multi-model: route each request to the cheapest model that passes, keep the expensive one for the genuinely hard reasoning.

The clearest signal of where this is headed is that Stripe just agreed to acquire OpenRouter (announced August 19, terms undisclosed, reported north of $7B by Bloomberg and TechCrunch), the router that sits in front of 400+ models and meters the tokens. When a payments company pays billions for the thing that routes and bills model calls, tokens have become a commodity you buy by the unit.

Commodities sold through a broker that shops for the cheapest passing option only get cheaper. If your business model assumes today’s token prices, plan for them falling by an order of magnitude.

The Plateau Comes Sooner Than the Roadmaps Say

Model progress will plateau, and given how fast the curve has moved, it will be sooner than you think. When the frontier stops jumping, the game stops being about the “smartest model” and becomes “same quality for the least money, power, and space,” which is a hardware question.

Today the answer is Nvidia H100 and H200 GPUs, and the challenge is two things: programmability (CUDA, a chip that runs whatever you invent next quarter) and networking (NVLink and InfiniBand lash thousands of GPUs into one fabric). But a frozen architecture is exactly what you burn into silicon, and plenty of companies are already betting on that.

Companies Already Building Fixed-Function AI Silicon

Companies Already Building Fixed-Function AI Silicon

Etched is the purest bet: Sohu hard-codes the transformer into silicon and runs nothing else. Its throughput claims are marketing until someone benchmarks them independently, but the thesis holds. If the transformer is the architecture for the next decade, a chip that does only transformers wins on cost per token by a margin a flexible chip can’t answer.

We’ve seen this arc in crypto mining: CPU, then GPU, then ASIC, each stage wiping out the last. Litecoin’s Scrypt was deliberately designed to be “ASIC-resistant” to keep mining on GPUs. It didn’t work; Bitmain shipped the Antminer L3, a Scrypt ASIC, and the holdouts were finished.

AI is harder than a hash, so this runs slower and messier, and Nvidia’s networking outlasts its CUDA one. But the people betting GPUs stay central forever are standing where the Scrypt holdouts stood. It’s a question of when, and the when is closer than the capex suggests.

What Happens to the Buildings

Meanwhile the industry is pouring concrete for a frontier that keeps advancing and a GPU that stays central.

What's Been Announced (proposed data centers)

What’s Been Announced (proposed data centers)

I don’t believe most of this gets built: Morgan Stanley pegs the data-center financing gap through 2028 at around $1.5T, Bain figures the industry is roughly $800B a year short on the revenue to fund it, and Satya Nadella has already said “there will be an overbuild.” Microsoft has been quietly canceling leases, which TD Cowen read as oversupply.

Now layer on the hardware shift. These campuses sit where power is cheap and land is empty, far from people, which is fine for training because training doesn’t care about latency. Inference does; it wants to sit near users.

So picture the end state: progress plateaus, the workload tilts to efficient inference on purpose-built silicon, and that inference wants a metro. What happens to a two-gigawatt training barn in rural Louisiana, built for GPUs doing a job that moved somewhere else? My honest guess is that some become the abandoned malls of this decade, stranded too far from anyone to repurpose. The ones near grid capacity, fiber, and people convert fine. The rest hold hardware that lost on cost per token, in a place nobody needs.

This is the ordinary shape of an infrastructure boom, and AI keeps working fine right through it. The first wave overbuilds general-purpose capacity chasing a moving frontier; the workload commoditizes and specializes; the hardware goes from flexible to fixed; and the winners are whoever got cost per useful token lowest, not whoever had the biggest model.

We ran this loop in mining, and in the fiber glut before it. Worth positioning for the back half now, while everyone else builds for the front.