DeepSeek Coder

1. Introduction of DeepSeek Coder

DeepSeek Coder is composed of a series of code language models, each trained from scratch on 2T tokens, with a composition of 87% code and 13% natural language in both English and Chinese. We provide various sizes of the code model, ranging from 1B to 33B versions. Each model is pre-trained on project-level code co... See more

pair-preference-model-LLaMA3-8B by RLHFlow: Really strong reward model, trained to take in two inputs at once, which is the top open reward model on RewardBench (beating one of Cohere’s).

DeepSeek-V2 by deepseek-ai (21B active, 236B total param.): Another strong MoE base model from the DeepSeek team. Some people are questioning the very high MMLU sc... See more

RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text

documents coming from 84 CommonCrawl snapshots and processed using

the CCNet pipeline. Out of these, there are 30B documents in the corpus

that additionally come with quality signals. In addition, we also provide the ids of duplicated documents whic... See more