DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI company “committed to making AGI a truth” and open-sourcing all its models. They started in 2023, but have been making waves over the previous month or two, and particularly this past week with the release of their 2 latest reasoning designs: DeepSeek-R1-Zero and the advanced DeepSeek-R1, likewise referred to as DeepSeek Reasoner.

They’ve released not only the models but likewise the code and assessment prompts for public use, together with an in-depth paper outlining their technique.

Aside from producing 2 extremely performant models that are on par with OpenAI’s o1 model, the paper has a great deal of valuable details around support learning, chain of thought reasoning, prompt engineering with reasoning models, and more.

We’ll start by focusing on the training process of DeepSeek-R1-Zero, which distinctively relied entirely on support knowing, rather of conventional supervised learning. We’ll then proceed to DeepSeek-R1, how it’s reasoning works, and some prompt engineering finest practices for thinking models.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s latest design release and comparing it with OpenAI’s thinking designs, particularly the A1 and A1 Mini designs. We’ll explore their training procedure, reasoning abilities, and some crucial insights into timely engineering for thinking models.

DeepSeek is a Chinese-based AI business dedicated to open-source advancement. Their current release, the R1 reasoning model, is groundbreaking due to its open-source nature and innovative training approaches. This consists of open access to the designs, prompts, and research study documents.

Released on January 20th, DeepSeek’s R1 attained outstanding performance on various criteria, measuring up to OpenAI’s A1 designs. Notably, they likewise released a precursor design, R10, which acts as the structure for R1.

Training Process: R10 to R1

R10: This design was trained specifically using reinforcement knowing without monitored fine-tuning, making it the very first open-source model to accomplish high efficiency through this approach. Training involved:

– Rewarding correct responses in deterministic jobs (e.g., math problems).
– Encouraging structured reasoning outputs utilizing templates with “” and “” tags

Through thousands of versions, R10 developed longer thinking chains, self-verification, and even reflective behaviors. For example, throughout training, the model demonstrated “aha” minutes and self-correction habits, which are rare in standard LLMs.

R1: Building on R10, R1 included numerous enhancements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human preference positioning for refined actions.
– Distillation into smaller designs (LLaMA 3.1 and 3.3 at different sizes).

Performance Benchmarks

DeepSeek’s R1 design performs on par with OpenAI’s A1 designs throughout lots of thinking benchmarks:

Reasoning and Math Tasks: R1 competitors or surpasses A1 models in precision and depth of reasoning.
Coding Tasks: A1 designs typically carry out much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 often outmatches A1 in structured QA jobs (e.g., 47% precision vs. 30%).

One noteworthy finding is that longer reasoning chains typically enhance performance. This lines up with insights from Microsoft’s Med-Prompt structure and OpenAI’s observations on test-time calculate and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some limitations:

– Mixing English and Chinese actions due to a lack of supervised fine-tuning.
– Less polished actions compared to chat models like OpenAI’s GPT.

These problems were resolved throughout R1’s improvement process, consisting of monitored fine-tuning and human feedback.

Prompt Engineering Insights

A fascinating takeaway from DeepSeek’s research study is how few-shot prompting abject R1’s efficiency compared to zero-shot or succinct customized prompts. This lines up with findings from the Med-Prompt paper and OpenAI’s suggestions to restrict context in thinking models. Overcomplicating the input can overwhelm the model and lower precision.

DeepSeek’s R1 is a significant action forward for open-source thinking designs, showing abilities that equal OpenAI’s A1. It’s an exciting time to experiment with these designs and their chat user interface, which is totally free to utilize.

If you have concerns or wish to discover more, take a look at the resources connected listed below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only technique

DeepSeek-R1-Zero sticks out from most other modern designs since it was trained using only support learning (RL), no monitored fine-tuning (SFT). This challenges the current standard method and opens brand-new chances to train thinking models with less human intervention and effort.

DeepSeek-R1-Zero is the very first open-source design to verify that sophisticated thinking capabilities can be established simply through RL.

Without pre-labeled datasets, the design finds out through experimentation, improving its habits, criteria, and weights based exclusively on feedback from the options it creates.

DeepSeek-R1-Zero is the base design for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero involved presenting the design with various thinking tasks, ranging from math issues to abstract reasoning challenges. The design created outputs and was examined based on its efficiency.

DeepSeek-R1-Zero got feedback through a benefit system that helped guide its knowing process:

Accuracy benefits: Evaluates whether the output is appropriate. Used for when there are deterministic outcomes (math problems).

Format rewards: Encouraged the design to structure its reasoning within and tags.

Training timely template

To train DeepSeek-R1-Zero to produce structured chain of thought sequences, the researchers utilized the following timely training design template, replacing timely with the thinking concern. You can access it in PromptHub here.

This template prompted the model to explicitly describe its thought procedure within tags before providing the last answer in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero began to produce advanced thinking chains.

Through thousands of training actions, DeepSeek-R1-Zero developed to resolve increasingly intricate problems. It learned to:

– Generate long reasoning chains that made it possible for deeper and more structured analytical

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own mistakes, showcasing emergent self-reflective habits.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is mostly a precursor to DeepSeek-R1, it still achieved high performance on numerous standards. Let’s dive into some of the experiments ran.

Accuracy enhancements throughout training

– Pass@1 accuracy started at 15.6% and by the end of the training it improved to 71.0%, similar to OpenAI’s o1-0912 design.

– The red solid line represents performance with bulk voting (comparable to ensembling and self-consistency strategies), which increased precision even more to 86.7%, going beyond o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s efficiency across multiple thinking datasets against OpenAI’s thinking models.

AIME 2024: 71.0% Pass@1, a little below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much even worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll look at how the reaction length increased throughout the RL training procedure.

This graph reveals the length of from the model as the training process advances. Each “step” represents one cycle of the model’s knowing process, where feedback is offered based upon the output’s efficiency, examined using the prompt template talked about earlier.

For each question (representing one action), 16 reactions were sampled, and the typical accuracy was computed to guarantee steady assessment.

As training progresses, the design generates longer reasoning chains, enabling it to resolve increasingly intricate reasoning tasks by leveraging more test-time compute.

While longer chains don’t always ensure much better outcomes, they normally correlate with improved performance-a pattern also observed in the MEDPROMPT paper (find out more about it here) and in the original o1 paper from OpenAI.

Aha moment and self-verification

One of the coolest aspects of DeepSeek-R1-Zero’s development (which likewise uses to the flagship R-1 model) is just how great the design ended up being at reasoning. There were advanced reasoning habits that were not explicitly configured however occurred through its reinforcement finding out process.

Over thousands of training actions, the design started to self-correct, review problematic logic, and validate its own solutions-all within its chain of thought

An example of this noted in the paper, described as a the “Aha minute” is below in red text.

In this instance, the design actually stated, “That’s an aha moment.” Through DeepSeek’s chat function (their version of ChatGPT) this kind of thinking typically emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and difficulties in DeepSeek-R1-Zero

While DeepSeek-R1-Zero had the ability to perform at a high level, there were some disadvantages with the design.

Language blending and coherence problems: The model sometimes produced actions that combined languages (Chinese and English).

Reinforcement knowing compromises: The lack of monitored fine-tuning (SFT) suggested that the model lacked the refinement needed for totally polished, human-aligned outputs.

DeepSeek-R1 was developed to attend to these problems!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning model from the Chinese AI lab DeepSeek. It develops on DeepSeek-R1-Zero, which was trained totally with reinforcement learning. Unlike its predecessor, DeepSeek-R1 includes monitored fine-tuning, making it more refined. Notably, it outshines OpenAI’s o1 model on numerous benchmarks-more on that later.

What are the main distinctions in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 builds on the foundation of DeepSeek-R1-Zero, which serves as the base design. The two differ in their training techniques and total efficiency.

1. Training approach

DeepSeek-R1-Zero: Trained entirely with reinforcement learning (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes monitored fine-tuning (SFT) first, followed by the exact same reinforcement finding out process that DeepSeek-R1-Zero damp through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Battled with language blending (English and Chinese) and readability issues. Its reasoning was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong thinking design, sometimes beating OpenAI’s o1, however fell the language blending concerns lowered use significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on the majority of reasoning benchmarks, and the reactions are far more polished.

In other words, DeepSeek-R1-Zero was an evidence of concept, while DeepSeek-R1 is the fully optimized version.

How DeepSeek-R1 was trained

To take on the readability and coherence issues of R1-Zero, the scientists integrated a cold-start fine-tuning phase and a multi-stage training pipeline when developing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a high-quality dataset of long chains of idea examples for initial supervised fine-tuning (SFT). This data was gathered using:- Few-shot prompting with comprehensive CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 went through the exact same RL procedure as DeepSeek-R1-Zero to refine its reasoning capabilities further.

Human Preference Alignment:

– A secondary RL phase enhanced the design’s helpfulness and harmlessness, guaranteeing better positioning with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking abilities were distilled into smaller, efficient models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark performance

The researchers evaluated DeepSeek R-1 throughout a variety of standards and against leading designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into numerous classifications, revealed listed below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied across all designs:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 outshined o1, Claude 3.5 Sonnet and other designs in the bulk of reasoning benchmarks.

o1 was the best-performing design in four out of the 5 coding-related benchmarks.

– DeepSeek carried out well on creative and long-context job job, like AlpacaEval 2.0 and ArenaHard, exceeding all other designs.

Prompt Engineering with reasoning designs

My favorite part of the post was the researchers’ observation about DeepSeek-R1’s sensitivity to prompts:

This is another datapoint that aligns with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft’s research study on their MedPrompt structure. In their study with OpenAI’s o1-preview model, they discovered that overwhelming reasoning designs with few-shot context broken down performance-a sharp contrast to non-reasoning designs.

The crucial takeaway? Zero-shot triggering with clear and succinct guidelines appear to be best when using thinking models.