700 Tokens Per Second: Meta’s New AI Chips Challenge Nvidia’s Grip

For years, the debate in the artificial intelligence industry has centered on a single question: can anyone break Nvidia’s stranglehold on the chips that power modern AI? The answer appears to be shifting. Google’s custom tensor processing units, or TPUs, are now running production workloads at Meta. This is not just a technical tweak; it is a signal that the most dominant AI player in social media is betting on a rival’s hardware to keep its systems running.

The move was announced via a new, PyTorch-native version of the SGLang inference engine. Until now, running these chips required using Google’s own JAX framework, a barrier that kept many developers away. The new software allows the hardware to work natively with PyTorch, the standard framework for building AI models, largely without rewriting code. Meta’s PyTorch-TPU lead, Zongwei Zhou, confirmed that the system is live in production today, citing a tight collaboration between Meta, Google, and the SGLang team at RadixArk, a startup founded by the engine’s creators.

Meta has been vocal about its need for flexibility. Jana van Greunen, a director of PyTorch engineering at Meta, stated that the company cannot be constrained by a single hardware architecture. By adopting this new stack, Meta can deploy its PyTorch workloads directly onto Google’s TPUs. Google co-developed this integration, known as TorchTPU, specifically with Meta’s engineering team to ensure a smooth transition.

This is not the first time the two giants have crossed paths on this hardware. Google’s older bridge, PyTorch/XLA, has been described as a predecessor to the new system. Reports last year indicated that Meta’s research teams used TPUs through that older bridge between 2020 and 2023, though those contracts were eventually canceled. The current push seems aimed squarely at Meta, which had previously resisted migrating entirely to Google’s JAX framework.

The physical location of these chips remains a bit of a mystery. While it is not yet clear exactly where Meta’s production TPUs are housed, industry talks from last year suggested a timeline involving Google Cloud TPU rentals in 2026 and potential purchases in 2027. Meta is reportedly signed on for a multi-year rental agreement, suggesting a long-term commitment rather than a short-term test.

The timing of this announcement is notable. It came just one day after Amazon revealed it was open-sourcing TorchNeuron, a similar backend for its own Trainium chips. Both moves appear designed to loosen the software lock-in that has benefited Nvidia, giving companies more options for where and how they run their models.

On the performance side, the numbers are impressive. A tuned version of vLLM, another inference engine now supporting TPUs, reportedly achieves over 700 tokens per second for a single user on the seventh-generation Ironwood TPU. This figure comes from Inferact, the startup behind vLLM. In comparison, a similar setup on Nvidia’s GB200 chips reached 452 tokens per second. At smaller batch sizes, the TPU setup delivers roughly 1.4 to two times the throughput of the Nvidia baseline when speculative decoding is not used.

Looking ahead, the SGLang-TorchTPU team plans to add support for more models, starting with DeepSeek V4.1 Flash and Reflection Beam. They are also working toward supporting new open models on their release day, a goal they aim to achieve by the first quarter of 2027. Google, Meta, and RadixArk plan to share more details at the SGLang Summit in San Francisco in November.