The LPU Inference Engine — fastest AI inference at 1/10th the energy
HQ: Mountain View, CA, United States | Founded: 2016 | Employees: 250-500 | Stage: Late Stage VC | Website: https://groq.com
Groq is an AI hardware and cloud inference company founded in 2016 by Jonathan Ross — the Google engineer who invented the Tensor Processing Unit (TPU) — along with Douglas Wightman (former X lab) and a team from the original TPU program. Headquartered in Mountain View, California, Groq developed the Language Processing Unit (LPU) Inference Engine based on its proprietary Tensor Streaming Processor (TSP) architecture. Unlike GPUs that rely on parallel cores and dynamic runtime scheduling, Groq's single-core, deterministic design with ~230MB on-chip SRAM and ~80 TB/s bandwidth executes workloads scheduled entirely at compile time, eliminating cache misses and pipeline stalls to deliver predictable, ultra-low latency inference for large language models such as Meta Llama 3.1, Mistral Mixtral, and OpenAI Whisper at hundreds of tokens per second.
The company monetizes via two vectors: GroqCloud, a developer API platform offering pay-per-token access to LPU-accelerated open models (with GroqChat playground), and GroqRack, a 9-server on-prem appliance requiring minimal external networking and data-center upgrades. Backed by BlackRock, Tiger Global, D1 Capital, Neuberger Berman, Social Capital, Addition, Cisco Investments, KDDI and Samsung Catalyst Fund, Groq raised $300M in April 2021 at ~$1.1B valuation and $640M Series D in August 2024 at $2.8B. With a $1.5B Aramco Digital/Humain partnership to build a Dammam data center and expand across Saudi Arabia, Groq is scaling its global data-center network to compete in the $100B+ AI accelerator market dominated by Nvidia.
Groq has emerged as Nvidia's most credible challenger in AI inference by betting on determinism over parallelism. Its LPU + Tensor Streaming Processor (TSP) architecture eliminates runtime scheduling overhead via compile-time orchestration, delivering predictable sub-millisecond latency and >100k tokens/sec on GroqCloud. With $640M Series D at $2.8B (BlackRock, Aug 2024), partnerships with Aramco Digital/Humain in Saudi Arabia for the world's largest LPU inference center, and 356,000+ GroqCloud developers, Groq is positioned to capture share of the $100B AI chip and $400B inference markets — though it faces foundry dependency, Nvidia CUDA moat, and capital intensity risks.
Browse: Home | Blog | About | All Startups A-Z | View full profile on StartupWiki