of the DeepSeek-R1 mannequin despatched ripples throughout the worldwide AI neighborhood. It delivered breakthroughs on par with the reasoning fashions from Meta and OpenAI, reaching this in a fraction of the time and at a considerably decrease value.
Past the headlines and on-line buzz, how can we assess the mannequin’s reasoning talents utilizing acknowledged benchmarks?
Deepseek’s user interface makes it straightforward to discover its capabilities, however utilizing it programmatically gives deeper insights and extra seamless integration into real-world functions. Understanding easy methods to run such fashions domestically additionally supplies enhanced management and offline entry.
On this article, we discover easy methods to use Ollama and OpenAI’s simple-evals to guage the reasoning capabilities of DeepSeek-R1’s distilled fashions based mostly on the well-known GPQA-Diamond benchmark.
Contents
(1) What are Reasoning Models?
(2) What is DeepSeek-R1?
(3) Understanding Distillation and DeepSeek-R1 Distilled Models
(4) Selection of Distilled Model
(5) Benchmarks for Evaluating Reasoning
(6) Tools Used
(7) Results of Evaluation
(8) Step-by-Step Walkthrough
Right here is the link to the accompanying GitHub repo for this text.
(1) What are Reasoning Fashions?
Reasoning fashions, reminiscent of DeepSeek-R1 and OpenAI’s o-series fashions (e.g., o1, o3), are massive language fashions (LLMs) skilled utilizing reinforcement studying to carry out reasoning.
Reasoning fashions suppose earlier than they reply, producing a protracted inside chain of thought earlier than responding. They excel in complicated problem-solving, coding, scientific reasoning, and multi-step planning for agentic workflows.
(2) What’s DeepSeek-R1?
DeepSeek-R1 is a state-of-the-art open-source LLM designed for superior reasoning, launched in January 2025 within the paper “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”.
The mannequin is a 671-billion-parameter LLM skilled with in depth use of reinforcement studying (RL), based mostly on this pipeline:
- Two reinforcement levels geared toward discovering improved reasoning patterns and aligning with human preferences
- Two supervised fine-tuning levels serving because the seed for the mannequin’s reasoning and non-reasoning capabilities.
To be exact, DeepSeek skilled two fashions:
- The primary mannequin, DeepSeek-R1-Zero, a reasoning mannequin skilled with reinforcement studying, generates information for coaching the second mannequin, DeepSeek-R1.
- It achieves this by producing reasoning traces, from which solely high-quality outputs are retained based mostly on their closing outcomes.
- It signifies that, not like most fashions, the RL examples on this coaching pipeline should not curated by people however generated by the mannequin.
The result is that the mannequin achieved efficiency akin to main fashions like OpenAI’s o1 model throughout duties reminiscent of arithmetic, coding, and complicated reasoning.
(3) Understanding Distillation and DeepSeek-R1’s Distilled Fashions
Alongside the complete mannequin, additionally they open-sourced six smaller dense fashions (additionally named DeepSeek-R1) of various sizes (1.5B, 7B, 8B, 14B, 32B, 70B), distilled from DeepSeek-R1 based mostly on Qwen or Llama as the bottom mannequin.
Distillation is a way the place a smaller mannequin (the “scholar”) is skilled to copy the efficiency of a bigger, extra highly effective pre-trained mannequin (the “trainer”).
On this case, the trainer is the 671B DeepSeek-R1 mannequin, and the scholars are the six fashions distilled utilizing these open-source base fashions:
DeepSeek-R1 was used because the trainer mannequin to generate 800,000 coaching samples, a mixture of reasoning and non-reasoning samples, for distillation through supervised fine-tuning of the bottom fashions (1.5B, 7B, 8B, 14B, 32B, and 70B).
So why will we do distillation within the first place?
The aim is to switch the reasoning talents of bigger fashions, reminiscent of DeepSeek-R1 671B, into smaller, extra environment friendly fashions. This empowers the smaller fashions to deal with complicated reasoning duties whereas being sooner and extra resource-efficient.
Moreover, DeepSeek-R1 has an enormous variety of parameters (671 billion), making it difficult to run on most consumer-grade machines.
Even essentially the most highly effective MacBook Professional, with a most of 128GB of unified reminiscence, is insufficient to run a 671-billion-parameter mannequin.
As such, distilled fashions open up the potential of being deployed on gadgets with restricted computational assets.
Unsloth achieved a powerful feat by quantizing the unique 671B-parameter DeepSeek-R1 mannequin down to only 131GB — a exceptional 80% discount in dimension. Nevertheless, a 131GB VRAM requirement stays a big hurdle.
(4) Collection of Distilled Mannequin
With six distilled mannequin sizes to select from, choosing the correct one largely will depend on the capabilities of the native system {hardware}.
For these with high-performance GPUs or CPUs and a necessity for optimum efficiency, the bigger DeepSeek-R1 fashions (32B and up) are ideally suited — even the quantized 671B model is viable.
Nevertheless, if one has restricted assets or prefers faster technology instances (as I do), the smaller distilled variants, reminiscent of 8B or 14B, are a greater match.
For this mission, I will likely be utilizing the DeepSeek-R1 distilled Qwen-14B mannequin, which aligns with the {hardware} constraints I confronted.
(5) Benchmarks for Evaluating Reasoning
LLMs are sometimes evaluated utilizing standardized benchmarks that assess their efficiency throughout numerous duties, together with language understanding, code technology, instruction following, and query answering. Widespread examples embody MMLU, HumanEval, and MGSM.
To measure an LLM’s capability for reasoning, we’d like tougher, reasoning-heavy benchmarks that transcend surface-level duties. Listed here are some standard examples centered on evaluating superior reasoning capabilities:
(i) AIME 2024 — Competitors Math
- The American Invitational Mathematics Examination (AIME) 2024 serves as a robust benchmark for evaluating an LLM’s mathematical reasoning capabilities.
- It’s a difficult math contest with complicated, multi-step issues that take a look at an LLM’s potential to interpret intricate questions, apply superior reasoning, and carry out exact symbolic manipulation.
(ii) Codeforces — Competitors Code
- The Codeforces Benchmark evaluates an LLM’s reasoning potential utilizing actual aggressive programming issues from Codeforces, a platform identified for algorithmic challenges.
- These issues take a look at an LLM’s capability to grasp complicated directions, carry out logical and mathematical reasoning, plan multi-step options, and generate right, environment friendly code.
(iii) GPQA Diamond — PhD-Stage Science Questions
- GPQA-Diamond is a curated subset of the most tough questions from the broader GPQA (Graduate-Level Physics Question Answering) benchmark, particularly designed to push the bounds of LLM reasoning in superior PhD-level subjects.
- Whereas GPQA features a vary of conceptual and calculation-heavy graduate questions, GPQA-Diamond isolates solely essentially the most difficult and reasoning-intensive ones.
- It’s thought of Google-proof, which means that they’re tough to reply even with unrestricted net entry.
- Right here is an instance of a GPQA-Diamond query:
On this mission, we use GPQA-Diamond because the reasoning benchmark, as OpenAI and DeepSeek used it to guage their reasoning fashions.
(6) Instruments Used
For this mission, we primarily use Ollama and OpenAI’s simple-evals.
(i) Ollama
Ollama is an open-source device that simplifies operating LLMs on our laptop or a neighborhood server.
It acts as a supervisor and runtime, dealing with duties reminiscent of downloads and atmosphere setup. This enables customers to work together with these fashions with out requiring a relentless web connection or counting on cloud providers.
It helps many open-source LLMs, together with DeepSeek-R1, and is cross-platform appropriate with macOS, Home windows, and Linux. Moreover, it gives an easy setup with minimal fuss and environment friendly useful resource utilization.
Vital: Guarantee your native system has GPU entry for Ollama, as this dramatically accelerates efficiency and makes subsequent benchmarking workout routines rather more environment friendly as in comparison with CPU. Run
nvidia-smiin terminal to examine if GPU is detected.
(ii) OpenAI simple-evals
simple-evals is a light-weight library designed to guage language fashions utilizing a zero-shot, chain-of-thought prompting strategy. It contains well-known benchmarks like MMLU, MATH, GPQA, MGSM, and HumanEval, aiming to mirror sensible utilization eventualities.
A few of you could find out about OpenAI’s extra well-known and complete analysis library known as Evals, which is distinct from simple-evals.
In actual fact, the README of simple-evals additionally particularly signifies that it’s not meant to exchange the Evals library.
So why are we utilizing simple-evals?
The easy reply is that simple-evals comes with built-in analysis scripts for the reasoning benchmarks we’re concentrating on (reminiscent of GPQA), that are lacking in Evals.
Moreover, I didn’t discover every other instruments or platforms, aside from simple-evals, that present an easy, Python-native technique to run quite a few key benchmarks, reminiscent of GPQA, notably when working with Ollama.
(7) Outcomes of Analysis
As a part of the analysis, I chosen 20 random questions from the GPQA-Diamond 198-question set for the 14B distilled mannequin to work on. The overall time taken was 216 minutes, which is ~11 minutes per query.
The result was admittedly disappointing, because it scored solely 10%, far beneath the reported 73.3% rating for the 671B DeepSeek-R1 mannequin.
The principle concern I observed is that in its intensive inside reasoning, the mannequin usually both failed to provide any reply (e.g., returning reasoning tokens as the ultimate traces of output) or offered a response that didn’t match the anticipated multiple-choice format (e.g., Reply: A).
As proven above, many outputs ended up as None as a result of the regex logic in simple-evals couldn’t detect the anticipated reply sample within the LLM response.
Whereas the human-like reasoning logic was attention-grabbing to look at, I had anticipated stronger efficiency when it comes to question-answering accuracy.
I’ve additionally seen on-line customers point out that even the bigger 32B mannequin doesn’t carry out in addition to o1. This has raised doubts concerning the utility of distilled reasoning fashions, particularly once they battle to offer right solutions regardless of producing lengthy reasoning.
That mentioned, GPQA-Diamond is a extremely difficult benchmark, so these fashions might nonetheless be helpful for less complicated reasoning duties. Their decrease computational calls for additionally make them extra accessible.
Moreover, the DeepSeek workforce really useful conducting a number of checks and averaging the outcomes as a part of the benchmarking course of — one thing I omitted because of time constraints.
(8) Step-by-Step Walkthrough
At this level, we’ve coated the core ideas and key takeaways.
Should you’re prepared for a hands-on, technical walkthrough, this part supplies a deep dive into the inside workings and step-by-step implementation.
Take a look at (or clone) the accompanying GitHub repo to observe alongside. The necessities for the digital atmosphere setup may be discovered here.
(i) Preliminary Setup — Ollama
We start by downloading Ollama. Go to the Ollama download page, choose your working system, and observe the corresponding set up directions.
As soon as set up is full, launch Ollama by double-clicking the Ollama app (for Home windows and macOS) or operating ollama serve within the terminal.
(ii) Preliminary Setup — OpenAI simple-evals
The setup of simple-evals is comparatively distinctive.
Whereas simple-evals presents itself as a library, the absence of __init__.py recordsdata within the repository means it’s not structured as a correct Python bundle, resulting in import errors after cloning the repo domestically.
Since it is usually not printed to PyPI and lacks commonplace packaging recordsdata like setup.py or pyproject.toml, it can’t be put in through pip.
Happily, we will make the most of Git submodules as an easy workaround.
A Git submodule lets us embody contents of one other Git repository inside our personal mission. It pulls the recordsdata from an exterior repo (e.g., simple-evals), however retains its historical past separate.
You possibly can select one in every of two methods (A or B) to tug the simple-evals contents:
(A) If You Cloned My Venture Repo
My mission repo already contains simple-evals as a submodule, so you possibly can simply run:
git submodule replace --init --recursive(B) If You’re Including It to a Newly Created Venture
To manually add simple-evals as a submodule, run this:
git submodule add https://github.com/openai/simple-evals.git simple_evalsWord: The simple_evals on the finish of the command (with an underscore) is essential. It units the folder title, and utilizing a hyphen as an alternative (i.e., easy–evals) can result in import points later.
Ultimate Step (For Each Strategies)
After pulling the repo contents, it’s essential to create an empty __init__.py within the newly created simple_evals folder in order that it’s importable as a module. You possibly can create it manually, or use the next command:
contact simple_evals/__init__.py(iii) Pull DeepSeek-R1 mannequin through Ollama
The following step is to domestically obtain the distilled mannequin of your alternative (e.g., 14B) utilizing this command:
ollama pull deepseek-r1:14bThe listing of DeepSeek-R1 fashions accessible on Ollama may be discovered here.
(iv) Outline configuration
We outline the parameters in a configuration YAML file, as proven beneath:
The mannequin temperature is ready to 0.6 (versus the standard default worth of 0). This follows DeepSeek’s utilization suggestions, which recommend a temperature vary of 0.5 to 0.7 (0.6 really useful) to forestall countless repetitions or incoherent outputs.
Do take a look at the apparently distinctive DeepSeek-R1 usage recommendations — particularly for benchmarking — to make sure optimum efficiency when utilizing DeepSeek-R1 fashions.
EVAL_N_EXAMPLES is the parameter for setting the variety of questions from the complete 198-question set to make use of for analysis.
(v) Arrange Sampler code
To assist Ollama-based language fashions throughout the simple-evals framework, we create a customized wrapper class named OllamaSampler saved inside utils/samplers/ollama_sampler.py.
On this context, a sampler is a Python class that generates outputs from a language mannequin based mostly on a given immediate.
Since current samplers in simple-evals solely cowl suppliers like OpenAI and Claude, we’d like a sampler class that gives a appropriate interface for Ollama.
The OllamaSampler extracts the GPQA query immediate, sends it to the mannequin with a specified temperature, and returns the plain textual content response.
The _pack_message technique is included to make sure the output format matches what the analysis scripts in simple-evals count on.
(vi) Create analysis run script
The next code units up the analysis execution in most important.py, together with the usage of the GPQAEval class from simple-evals to run GPQA benchmarking.
The run_eval() perform is a configurable analysis runner that checks LLMs via Ollama on benchmarks like GPQA.
It masses settings from the config file, units up the suitable analysis class from simple-evals, and runs the mannequin via a standardized analysis course of. It’s saved in most important.py, which may be executed with python most important.py.
Following the steps above, we have now efficiently arrange and executed the GPQA-Diamond benchmarking on the DeepSeek-R1 distilled mannequin.
Wrapping It Up
On this article, we showcased how we will mix instruments like Ollama and OpenAI’s simple-evals to discover and benchmark DeepSeek-R1’s distilled fashions.
The distilled fashions could not but rival the 671B parameter unique mannequin on difficult reasoning benchmarks like GPQA-Diamond. Nonetheless, they exhibit how distillation can increase entry to LLM reasoning capabilities.
Regardless of subpar scores in complicated PhD-level duties, these smaller variants could stay viable for much less demanding eventualities, paving the best way for environment friendly native deployment on a wider vary of {hardware}.
Earlier than you go
I welcome you to observe my GitHub and LinkedIn to remain up to date with extra partaking and sensible content material. In the meantime, have enjoyable benchmarking LLMs with Ollama and simple-evals!
