WebDataset

huggingface.co

RelatedInsightsHighlights

Thumbnail of www-x-com-tom-doerr-status-1964666664324043027-c6ecb57c4b8341ba

view and search huge datasets, preview archives, supports many formats https://t.co/lM6I5Y4ggW

Tom Dörr

x.com

A 24-trillion-token web dataset with document-level metadata just dropped on @huggingface License: apache-2.0 ESSENTIAL-WEB v1.0 collects 24 trillion tokens from Common Crawl. Each document is labeled with a 12-field taxonomy covering topic, page type, complexity, and quality .___LINEBREAK_... See more

Rohan Paul

x.com

An introduction to streaming on the web. Learn about chunking data, handling backpressure, server-side events, and more. https://t.co/txEctR4iUr

Vercel x.com