OpenAI’s progress from GPT-4 to Orion has been slower than expected, according to recent reports. Despite completing only 20% of Orion’s training, it has already shown comparable intelligence, task fulfillment, and question-answering abilities to GPT-4. While Orion has outperformed previous models, the improvement in quality is not as significant as the leap from GPT-3 to GPT-4. This has raised questions about whether advancements in language model scaling have reached a plateau. Notably, AI critic Gary Marcus was quick to declare victory, stating on X that “game over” for GPT. However, this may have been premature, as OpenAI researchers and the authors of the report have clarified that the progress of upcoming models has been misrepresented. They explain that there are now two key dimensions of scaling for models like the o1 series: training time and inference time. While traditional scaling laws still apply, the introduction of a second dimension has unlocked new capabilities. This is due to o1 being trained with reinforcement learning, allowing it to “think” before responding. This introduces a new dimension to scaling, as the model is no longer limited by pretraining and can also scale inference compute. OpenAI researcher Noam Brown further explains that o1’s “thinking” process is more authentic and robust compared to traditional AI models, as it engages in an internal reasoning process similar to how humans think. Instead of simply producing an answer, the model actively considers and evaluates options, leading to a more accurate and nuanced response. Overall, while the progress from GPT-4 to Orion may not be as dramatic as previous advancements, the introduction of a new scaling dimension has the potential to unlock even more impressive capabilities in the future.
