Polars — Processing hundreds of GBs of textual data on a daily basis at MDPI

The ability to implement custom Polars plugins in Rust is invaluable. Since we process a lot of textual data for our NLP applications, we can create optimized functions to clean text or detect a language, with data being processed efficiently in batches. This level of customization is rarely seen in other typical processing engines and is even impo... See more