China’s new DeepSeek Large Language Model (LLM) has disrupted the US-dominated market, offering a relatively high-performance chatbot model at a significantly lower cost.

The reduced cost of development and lower subscription prices compared with US AI tools contributed to American chip maker Nvidia losing US$600 billion (£480 billion) in market value over one day. Nvidia makes the computer chips used to train the majority of LLMs, the underlying technology used in ChatGPT and other AI chatbots. DeepSeek uses cheaper Nvidia H800 chips over the more expensive state-of-the-art versions.

ChatGPT developer OpenAI reportedly spent somewhere between US$100 million and US$1 billion on the development of a very recent version of its product called o1. In contrast, DeepSeek accomplished its training in just two months at a cost of US$5.6 million using a series of clever innovations.

But just how well does DeepSeek’s AI chatbot, R1, compare with other, similar AI tools on performance?

DeepSeek claims its models perform comparably to OpenAI’s offerings, even exceeding the o1 model in certain benchmark tests. However, benchmarks that use Massive Multitask Language Understanding (MMLU) tests evaluate knowledge across multiple subjects using multiple-choice questions. Many LLMs are trained and optimised for such tests, making them unreliable as true indicators of real-world performance.

An alternative methodology for the objective evaluation of LLMs uses a set of tests developed by researchers at Cardiff Metropolitan, Bristol and Cardiff universities – known collectively as the Knowledge Observation Group (KOG). These tests probe LLMs’ ability to mimic human language and knowledge through questions that require implicit human understanding to answer. The core tests are kept secret, to avoid LLM companies training their models for these tests.

KOG deployed public tests inspired by work by Colin Fraser, a data scientist at Meta, to evaluate DeepSeek against other LLMs. The following results were observed:

The tests used to produce this table are “adversarial” in nature. In other words, they are designed to be “hard” and to test LLMs in way that are not sympathetic to how they are designed. This means the performance of these models in this test is likely to be different to their performance in mainstream benchmarking tests.

DeepSeek scored 5.5 out of 6, outperforming OpenAI’s o1 – its advanced reasoning (known as “chain-of-thought”) model – as well as ChatGPT-4o, the free version of ChatGPT. But Deepseek was marginally outperformed by Anthropic’s ClaudeAI and OpenAI’s o1 mini, both of which scored a perfect 6/6. It’s interesting that o1 underperformed against its “smaller” counterpart, o1 mini.

DeepThink R1 – a chain-of-thought AI tool made by DeepSeek – underperformed in comparison to DeepSeek with a score of 3.5.

This result shows how competitive DeepSeek’s chatbot already is, beating OpenAI’s flagship models. It is likely to spur further development for DeepSeek, which now has a strong foundation to build upon. However, the Chinese tech company does have one serious problem the other LLMs do not: censorship.

Censorship challenges

Despite its strong performance and popularity, DeepSeek has faced criticism over its responses to politically sensitive topics in China. For instance, prompts related to Tiananmen Square, Taiwan, Uyghur Muslims and democratic movements are met with the response: “Sorry, that is beyond my current scope.”

But this issue is not necessarily unique to DeepSeek, and the potential for political influence and censorship in LLMs more generally is a growing concern. The announcement of Donald Trump’s US$500 billion Stargate LLM project, involving OpenAI, Nvidia, Oracle, Microsoft, and Arm, also raises fears of political influence.