LLMs

So right now, LLMs (Large Language Models) are all the rage. But in the future, it’s possible that the way we get things done is composing things with a combination of LLMs, SMMs (Small, Mighty Models), agents and tools.

It’s what I call Cognitive Composition (because it sounds cool and I have a longtime love affair with alliteration).

This is how we... See more

A new v0.4.0 release of lm-evaluation-harness is available !

New updates and features include:

• Internal refactoring


• Config-based task creation and configuration


• Easier import and sharing of externally-defined task config YAMLs


• Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource


• More advanced configuration

... See more

Easily chunk complex documents the same way a human would.

Chunking documents is a challenging task that underpins any RAG system. High quality results are critical to a sucessful AI application, yet most open-source libraries are limited in their ability to handle complex documents.

Open Parse is designed to fill this gap by providing a flexible,... See more

A solution is to self-host an open-sourced or custom fine-tuned LLM. Opting for a self-hosted model can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications. Considering self-hosted solutions requires weighing these different trade-offs carefully.

To train LLMs, you need data that is:

Large — Sufficiently large LMs require trillions of tokens.

Clean — Noisy data reduces performance.

Diverse — Data should come from different sources and different knowledge bases.

What does clean data look like?

You can de-duplicate data with simple heuristics. The most basic would be removing any exact duplicates... See more

Since we launched ChatGPT Enterprise a few months ago, early customers have expressed the desire for even more customization that aligns with their business. GPTs answer this call by allowing you to create versions of ChatGPT for specific use cases, departments, or proprietary datasets. Early customers like Amgen, Bain, and Square are already... See more

Amplify Partners was running a survey among 800+ AI engineers to bring transparency to the AI Engineering space. The report is concise, yet it provides a wealth of insights into the technologies and methods employed by companies for the implementation of AI products.

Highlights

👉 Top AI use cases are code intelligence, data extraction and workflow... See more