Nicolay Gerold

Data-Juicer: A One-Stop Data Processing System for Large Language Models

Data-Juicer is a one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs. This project is being actively updated and maintained, and we will periodically enhance and add more features and data recipes. We welcome you to join us in pro... See more

Hiring developers is a lot like building an NFL team.

You can draft fresh college graduates for a very low cost.

Not every college grad will make it to the big leagues.

Some will be superstars in the first week.

Your team needs to be a balance of college grads and veterans.

You can't afford a team of nothing but veterans.

You also don't need vetera... See more

𝗺𝗲𝘁𝗵𝗼𝗱𝘀 𝗼𝗳 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗮𝗻 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗟𝗟𝗠 𝗲𝘅𝗶𝘀t ↓

- 𝘊𝘰𝘯𝘵𝘪𝘯𝘶𝘦𝘥 𝘱𝘳𝘦-𝘵𝘳𝘢𝘪𝘯𝘪𝘯𝘨: utilize domain-specific data to apply the same pre-training process (next token prediction) on the pre-trained (base) model

- 𝘐𝘯𝘴𝘵𝘳𝘶𝘤𝘵𝘪𝘰𝘯 𝘧𝘪𝘯𝘦-𝘵𝘶𝘯𝘪𝘯𝘨: the pre-trained (base) model is fine-tuned on ... See more

With Quary, engineers can:

• 🔌 Connect to their Database


• 📖 Write SQL queries to transform, organize, and document tables in a database


• 📊 Create charts, dashboards and reports (in development)


• 🧪 Test, collaborate & refactor iteratively through version control


• 🚀 Deploy the organised, documented model back up to the database

View the documentation.

“Writing shows up within a company in two primary ways: paper trails and curations.

Paper trails are documented accounts of what happened, typically produced in the run of work while it’s happening. Meeting notes are the most obvious example.

Curations are artifacts of work produced to contribute to the system of shared knowledge. These are typically... See more

🔎 GPT Researcher

GPT Researcher is an autonomous agent designed for comprehensive online research on a variety of tasks.

The agent can produce detailed, factual and unbiased research reports, with customization options for focusing on relevant resources, outlines, and lessons. Inspired by the recent Plan-and-Solve and RAG (Retrieval Augmented Gene... See more

Both the OODA loop and PDCA (Plan, Do, Check, Act) cycle can be applied as frameworks for AI agents to go through retrospection phases or steps while performing tasks. The choice between the two depends on the specific context and the desired outcome of the retrospection process. Let's examine both frameworks in this context:

OODA loop:

Observe: The ... See more

WebDataset

WebDataset is a library for writing I/O pipelines for large datasets. Its sequential I/O and sharding features make it especially useful for streaming large-scale datasets to a DataLoader.

The WebDataset format

A WebDataset file is a TAR archive containing a series of data files. All successive data files with the same prefix are consider... See more