Ask a chatbot a question, and the model has to read your words, think, and type a reply. That whole act is what tech firms call inference, and it is the part of AI you actually feel when a response drags.
Training gets most of the attention, but inference is what runs every time someone opens the app. It is also where the costs pile up.
OpenAI thinks it has a chip that can handle this stage faster and cheaper. The company brought early proof to the Hot Chips conference on August 25, 2026, where it introduced Jalapeño, its custom inference chip.
OpenAI's hardware chief, Richard Ho, put it plainly. "The bottom line is that the results show a very, very significant performance advance over state of the art," he said.
He also explained the payoff: "Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It's very efficient to serve a lot of customers, but it can also be very low latency."
As AI benchmarks shift, your money can grow steadily with the free Always Be Buying E-Book
Most AI chips come from Nvidia, but OpenAI went its own way. It built Jalapeño with Broadcom, and it used its own models to help design the chip, a project first announced in October 2025.
OpenAI wants Jalapeño to be the first of many generations. OpenAI plans to design products, models, chips, and memory together so they work as one system instead of being patched together later.
The design chases specific bottlenecks: prefill, the part where the chip reads your prompt and starts organizing a response, and communication, the time spent moving data between parts of the system.
OpenAI also keeps important data close to the processor. The chip holds model state, including something called the KV cache, a short-term memory the model uses while writing a reply, which means less data has to travel and fewer delays pile up.
The company wrote in a blog post: "We designed Jalapeño to minimize data movement and communication delays."
The same post explained: "This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase."
The catch: Jalapeño's benchmark win came against an Nvidia Blackwell system. Competitors may have advanced by the time the chip actually ships, so the lead may not stay as wide as it looks today.
The deployment timeline gives the chip room to prove itself in real data centers. It also sets up a stretch where OpenAI runs its own chips next to the Nvidia systems it already uses.
For investors, the key is what faster, cheaper inference does to the AI business. Every ChatGPT-style request costs money to run, and inference is the stage that decides whether AI products can be profitable at a price people will pay.
If OpenAI can serve the same number of answers with less power, it can cut the cost of providing those answers. That kind of pressure has a way of reaching both the price you pay for AI tools and the companies in your portfolio.
When new chips steal the spotlight, remember the real win is the free Always Be Buying E-Book
