Grok 3 vs o3 Comparison

On February 17th, xAI’s Grok 3 made its debut into the world, to much fanfare. In a livestream demo, it was revealed that Grok 3 is being deployed with a reasoning model. So, it’s natural to want to compare it with other, similar reasoning models such as OpenAI’s o3. Both are successors to their previous iterations, promising leaps in performance and capability. Here's how they stack up against each other.

About Grok 3

Grok 3 was debuted in a livestream late in the evening of February 17th, 2025. According to a blog post on xAI's website, the model was trained using the Colossus supercomputer in Memphis, TN, and used 10 times the compute that its predecessor models did. Grok 3 adds reasoning capabilities, as well as improvements on the base models. The reasoning models of Grok 3 and Grok 3 mini were trained using reinforcement learning and were guided to follow a more human-like reasoning process. Training for both models is still ongoing, and Elon suggested users would notice improvements every day.

Performance on Benchmarks

When it comes to raw performance, both Grok 3 and o3 have been put through rigorous testing. Based on what was revealed in the livestream, xAI chose to share the following benchmark data for its base Grok 3 and Grok 3 mini models. NB: xAI included o4 data points but not o3 data points, so the o3 data points have been added in italics and sourced from this article comparing o3 and DeepSeek R1:

• Math (AIME ‘24) 
  • Grok 3 - 52
  • Grok 3 mini - 40
  • o3 - 96.7
• Science (GPQA)
  • Grok3 - 75
  • Grok 3 mini - 65
  • o3 - 87.7
• Coding (LCB Oct - Feb)
  • Grok 3 - 57
  • Grok 3 mini - 41 
  • o3 - no data point for this test

Performance on Benchmarks with Reasoning + More Test Time Compute

After explaining the virtues of more test time compute, representatives from xAI shared how their reasoning models did on the same benchmarks. With enhanced reasoning and test time compute, Grok 3 and Grok 3 mini perform much better in relation to o3 and even outperforms o3 mini high in math, science, and coding. Here, they directly compared their models to o3 mini high. Data points for o3 have again been added in italics:

• Math (AIME ‘24)
  • Grok 3 - 93
  • Grok 3 mini - 96
  • o3 - 96.7
  • o3 mini high - 87
• Math (AIME '25)
  • Grok 3 - 93
  • Grok 3 mini - 90
  • o3 - no data point for this benchmark
  • o3 mini high - 87
• Science (GPQA)
  • Grok 3 - 85
  • Grok 3 mini - 84 
  • o3 - 87.7
  • o3 mini high - 80
• Coding (LCB Oct - Feb)
  • Grok 3 - 79
  • Grok 3 mini - 80
  • o3 - no data point
  • o3 mini high - 74

Chatbot Arena - A New Benchmark

As part of their livestream, representatives from xAI shared the results of a benchmark they called Chatbot Arena (LMSYS) in which they pitted an early version of Grok 3 against various iterations of Gemini, DeepSeek, and o3 mini. In this test, users were presented with an interface that stripped away all identifiable markers of which models were which. Users would then submit a query and be presented with 2 model responses. According to xAI’s data points, the early version of Grok 3 outperformed all models it was pitted against, scoring above a 1400. Coming in second just above 1380 was Gemini 2.0-flash-thinking-exp-01-21. The exact metrics the preference rankings were based on or what kinds of queries were submitted were not revealed at the time of the livestream. They touted the benchmark’s usefulness as a blind test and a way to argue that the models were not simply regurgitating memorized answers. 

Reasoning Capabilities

Both models boast strong reasoning models, and each have their strengths. For Gork 3, representatives from xAI explained how Grok's approach to reasoning often involves a more human-like understanding of scenarios, particularly its ability to self-correct and check multiple possible solutions to a problem. As part of the demo, representatives also highlighted Grok's creative approach to problem solving when they had Grok create a video game that was a combination of Tetris and Bejeweled. Elon then took the opportunity to announce the launch of an AI video game studio through xAI. o3, on the other hand, shines in structured reasoning, touting their model’s excellence in coding and math, where it can systematically break down complex problems into manageable parts, offering step-by-step solutions that are particularly useful in educational contexts or technical troubleshooting.

Model Architecture and Innovations

Though the specifics of their internal architectures are kept tightly under wraps, it's clear that both models continue to push the boundaries of what LLMs are capable of. Both models appear to use some degree and variety of reinforcement learning and to emphasize the use of more test time compute if it suits the situation. Recently, OpenAI also announced that they were able to achieve a two order of magnitude cost reduction for their reasoning models. With the launch of the Colossus supercomputer in Memphis, and their newest data center in Atlanta, it will be interesting to see how xAI is able to use the enormous amount of compute at their disposal to improve its products while also becoming more efficient. Initially launched with 100,000 NVIDIA H100 GPUs, Colossus expanded to 200,000 GPUs and has plans to reach over 1 million GPUs.

User Interface and Access

Grok 3 can be accessed through both a user’s X account as well as through its website grok.com. Both models offer a way to toggle on different reasoning levels and functionalities. To access capabilities such as Deep Search (agentic search engine), Think (reasoning), and Big Brain (enhanced reasoning + more test time compute), users must access the model through the Grok website. As of this article's publication, Big Brain does not yet appear to be rolled out to users. o3 can be accessed through its website at chat.openai.com. o3 can be toggled from o3-mini to o3-mini-high. Users can also toggle Search (internet) and Deep Research (in-depth reasoning). Both models allow users to see a truncated version of each model’s reasoning process, and both models require subscription to a higher subscription tier to use their most advanced features. 

Conclusion

Choosing between Grok 3 and o3 depends heavily on the specific needs of your project or application. Grok 3 made a promising debut, demonstrating enormous strides in an incredibly short period of time, thanks in no small part to its Colossus supercomputer. o3 mini-high has some fierce competition to deal with now, but with the idea of a GPT 5 already generating buzz, it will be exciting to see what comes next. Both models represent the cutting edge of AI, each pushing the boundaries in their unique ways, promising exciting developments in the AI landscape.