Open-R1: a Fully Open Reproduction Of DeepSeek-R1

Hey there! This article is an introduction to the task, not a claim that we have actually replicated R1 yet. We’re developing in the open, so as quickly as we have assessment numbers, we’ll share them. You can follow our development on Hugging Face and GitHub.

True, however it appears like there’s nothing to be examined as of right now. I presume the ultimate objective is to train a brand-new thinking design and then utilize the exact same assessment metrics as o1 and the DeepSeek-R1.

Well, there need to be at least some peace of mind check and validation to ensure the model was trained correctly.

Oh yes, if you are talking about the of deepseek’s model it’s coming very quickly!

As mentioned in the blog site post there is no model called Open-R1 to test at all … not yet anyway. This is a blog site laying out that Hugging face will take the R1 Deepseek model, exercise how it was built as outlined in the paper and from what they released, and then replicate that procedure.

in fact this is quite much how science works … A comes up with a plan, discovery or development and it is checked by B, C and D to see if it is reproduceable. Thats been the cornerstone of research now for a couple of centuries.

This blog site is not stating they have actually currently done so … Its a blog site outlining an intent to begin training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was only launched recently, and even in their paper they described the compute hours needed. While those are low compute hours for a SOTA model this does not suggest you can train said design in a week. I ‘d personally love to be able to train a transformer design in a week, but we might require to wait a while for that level of calculate innovation.

So there are no benchmarks for a model that has not been constructed yet right? As described in the blog, and once again in reply to your concern.

However fear not, there is a GitHub Repo already and contributors (hell I may join myself), some prelim work done, and a plan of attack. A good starting position.

n
@edbeeching
has actually examined the launched models currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so jointly …/ s. This is what the new AI czars are saying

Hi! This post is an intro to the job, not a claim that we’ve recreated R1 yet. We will completely share the missing piece when we have them, you can expect the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s nice and crucial to comprehend this remarkable buzz that lacks technical comprehension and description. Science has to do with reproduction, and if they declare to be open, let them fullfill the open part.

Please do publish the training expense.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will indeed be striving to ensure this training recipe can work for small language models on consumer hardware considering that not everybody has a cluster of H100s at home:-RRB- The tool we used for the images was Excalidraw! https://excalidraw.com

looking forward to it! WTF are your discussing?

need to be a joke

It’s truly cool to see how the entire open source community comes together!

Ops …

5.5 M is number press reporter in the deepseekv3 tech report (just the training, not the experiment afaik), for R1 hard to estimate tbh but much less than 5.5 M imo

Historically, they have actually never launched code or datasets of their LLM training, so I would not expect this time to be various. If they would launch it that would be amazing naturally!

Yes naturally!

So essentially you’re asking to replace existing censorship with another flavour of censorship?

The code for the designs are inside the design repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and creator of EQUATOR. My research team will be dealing with a paper focused on replicating certain elements of DeepSeek R1. Our aim is to replicate the cold start and offer your group with a dataset that includes COT and other methods to support these efforts. We like to contribute our work to assist. Please let me know if you discover this useful. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the assessment numbers? without it you can’t call it recreation.

8 replies

True, but it looks like there’s nothing to be evaluated since right now. I assume the supreme goal is to train a new thinking model and then utilize the very same assessment metrics as o1 and the DeepSeek-R1.

That’s rather fascinating, I was asking myself why the concerns the author exposed here are not being asked by others? I think the work they have actually done is remarkable however at the same time I question why they wouldn’t put these missing pieces on if they are supposed to be fully open.
Why even without reproduction and understanding of the development they could impact so much the market in this method?

4 replies

Hi! This blog site post is an intro to the task, not a claim that we’ve replicated R1 yet. We will totally share the missing piece when we have them, you can expect the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is great that we see more effort into this direction: more optimization and less strength.
Also question what tool did the author usage for producing action diagram.

2 replies

Excalidraw I’m so grateful that initiative like this currently exist, I’m gon na attempt to contribute:-RRB- 1 reply

looking forward to it! So racist articel

2 replies

WTF are your discussing?

Awesome to have this open reproduction began!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s really cool to see how the entire open source community comes together!

Does anyone know the real training cost of r1? I can’t discover it in the paper or the statement post. Is the 6M expense reported by media simply the number drawn from v3’s training cost?

2 replies

Ops …

Has anyone asked the DeepSeek team to publish their training information and code, or at least share them independently with an independent replication task like this? Have they rejected such a request?

A loyal replication depends on using the exact same dataset and hyperparameters. Otherwise, any significant discrepancies with the published standards would be difficult to pin down-whether due to training data differences or the duplication approach itself.

1 reply

Historically, they have never released code or datasets of their LLM training, so I wouldn’t anticipate this time to be different. If they would release it that would be fantastic naturally!

In the meantime we have to make best guess price quotes and see if we can arrive ourselves.

You offer great replication process of Deepseek reasoning training. I will try something comparable to it.

This is really good information, can we fine tune with specific usage case when code is released?

1 reply

Yes of course!

Please consider eliminating prejudiced, tainted or unaligned training data and make an effort to eliminate copyrighted works from the crawl from consumption. This will make the design more functional. If you reused anthropic curation checks, this may likewise assist, remove obviouslybiased data will likely add a lot of worth. We don’t desire another polluted, unaligned open source model, right? And no corporate would ever use deepseek or a model that recycles it, right?
We value your work for the advantage of humanity, we hope.
Miike C from NJ

1 reply

So basically you’re asking to replace existing censorship with another flavour of censorship?

Can’t wait! Hopefully the model will be uncensored but whatever you can do is alright! Love seeing open source building itself up. I’m not wise sufficient to actually assist however I can contribute support lol

Hello guys, I am even simply looking for code for DeepSeek-V2, in order to fully comprehend multi-head hidden attention. You do not appear to have code in Hugging Face even for that. Or am I missing something? Don’t see anything in src/transformers/models. MLA is not correctly explained in their paper, so it would be necessary to have code for this.