DeepSeek Coder comprises a series of code language models trained from scratch on both 87% code and 13% natural language in English and Chinese, with each model pre-trained on 2T tokens. We provide various sizes of the code model, ranging from 1B to 33B versions. Each model is pre-trained on repo-level code corpus by employing a window size of 16K ... See more

Recently proposed open-source language models have placed an emphasis upon inference speed. Such work has shown us that inference speed can be improved by up to 5X (or more) by making some changes to the decoder-only transformer architecture. Here are three examples that have appeared frequently in recent work… ___LINEB... See more