OpenAI Said to Cut Inference Costs by Over Half via Software Alone

OpenAI engineers have found a software optimization for existing models that cuts inference costs by more than 50%, a move reportedly shared internally at the company. It is a pure software improvement requiring no additional hardware, according to a report by The Information that has been echoed by secondary coverage. When the optimization was applied to ChatGPT's logged-out users, or guest traffic, the number of required NVIDIA GPUs reportedly dropped to just a few hundred. As reported, the key point is that it was achieved without adding hardware.