LLMs
by Nicolay Gerold · updated 2mo ago
LLMs
- Large variety of ready-to-use LLM evaluation metrics (all with explanations) powered by
ANY
LLM of your choice, statistical methods, or NLP models that runs
locally on your machine
:- G-Eval
- Summarization
- Answer Relevancy
- Faithfulness
- Contextual Recall
- Contextual Precision
- RAGAS
- Hallucination
- Toxicity
- Bias
- etc.
from GitHub - confident-ai/deepeval: The LLM Evaluation Framework
Nicolay Gerold added 4mo ago
- Large variety of ready-to-use LLM evaluation metrics (all with explanations) powered by
- The multiple cantilevered AI overhangs:
Compute overhang. We have much more compute than we are using. Scale can go much further.
Idea overhang. There are many obvious research ideas and combinations of ideas that haven’t been tried in earnest yet.
Capability overhang. Even if we stopped all research now, it would take ten years to digest the new capa... See morefrom Shortwave — rajhesh.panchanadhan@gmail.com [Gmail alternative]
Nicolay Gerold added 10mo ago
- OpenGPTs
This is an open source effort to create a similar experience to OpenAI's GPTs. It builds upon LangChain, LangServe and LangSmith. OpenGPTs gives you more control, allowing you to configure:- The LLM you use (choose between the 60+ that LangChain offers)
- The prompts you use (use LangSmith to debug those)
- The tools you give it (choose from LangC
from langchain-ai/opengpts by langchain-ai
Nicolay Gerold added 1y ago
- DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Table of Contents- Introduction
- Key LLM Serving Techniques
- Dynamic SplitFuse: A Novel Prompt and Generation Composition Strategy
- Performance Evaluation
- DeepSpeed-FastGen: Implementation and Usage
- Try out DeepSpeed-FastGen
- Acknowledgements
1. Introduction
Large langu... See morefrom DeepSpeed-FastGen by microsoft
Nicolay Gerold added 1y ago
- In some applications, such as inline code suggestions, the best AI models are too expensive, so tools like Github Copilot use carefully tuned smaller models and various search heuristics to provide results. In other applications, even the largest models, like GPT-4, are too cheap!
from The Shift From Models to Compound AI Systems by Matei Zaharia, Omar Khattab, Lingjiao Chen, et al.
Nicolay Gerold added 7mo ago
- Self-play is the idea that an agent can improve its gameplay by playing against slightly different versions of itself because it’ll progressively encounter more challenging situations. In the space of LLMs, it is almost certain that the largest portion of self-play will look like AI Feedback rather than competitive processes.
from The Q* hypothesis: Tree-of-thoughts reasoning, process reward models, and supercharging synthetic data by Nathan Lambert
Nicolay Gerold added 10mo ago
- To train LLMs, you need data that is:
Large — Sufficiently large LMs require trillions of tokens.
Clean — Noisy data reduces performance.
Diverse — Data should come from different sources and different knowledge bases.
What does clean data look like?
You can de-duplicate data with simple heuristics. The most basic would be removing any exact duplicates ... See morefrom Shortwave — rajhesh.panchanadhan@gmail.com [Gmail alternative]
Nicolay Gerold added 10mo ago
- So right now, LLMs (Large Language Models) are all the rage. But in the future, it’s possible that the way we get things done is composing things with a combination of LLMs, SMMs (Small, Mighty Models), agents and tools.
It’s what I call Cognitive Composition (because it sounds cool and I have a longtime love affair with alliteration).
This is how we... See morefrom Shortwave — rajhesh.panchanadhan@gmail.com [Gmail alternative]
Nicolay Gerold added 10mo ago
- Systems can be dynamic. Machine learning models are inherently limited because they are trained on static datasets, so their “knowledge” is fixed. Therefore, developers need to combine models with other components, such as search and retrieval, to incorporate timely data. In addition, training lets a model “see” the whole training set, so more com
... See morefrom The Shift From Models to Compound AI Systems by Matei Zaharia, Omar Khattab, Lingjiao Chen, et al.
Nicolay Gerold added 7mo ago