LLMs

Amplify Partners was running a survey among 800+ AI engineers to bring transparency to the AI Engineering space. The report is concise, yet it provides a wealth of insights into the technologies and methods employed by companies for the implementation of AI products.

Highlights

👉 Top AI use cases are code intelligence, data extraction and workflow... See more

Some tasks are easier to improve via system design. While LLMs appear to follow remarkable scaling laws that predictably yield better results with more compute, in many applications, scaling offers lower returns-vs-cost than building a compound system. For example, suppose that the current best LLM can solve coding contest problems 30% of the

... See more

Google Deepmind used similar idea to make LLMs faster in Accelerating Large Language Model Decoding with Speculative Sampling. Their algorithm uses a smaller draft model to make initial guesses and a larger primary model to validate them. If the draft often guesses right, operations become faster, reducing latency.

There are some people speculating... See more

Study finds RLHF reduces LLM creativity and output variety : A new research paper posted in /r/LocalLLaMA shows that while alignment techniques like RLHF reduce toxic and biased content, they also limit the creativity of large language models, even in contexts unrelated to safety.

Deploying a Generative AI model requires more than a VM with a GPU. It normally includes:

• Container Service : Most often Kubernetes to run LLM Serving solutions like Hugging Face Text Generation Inference or vLLM.


• Compute Resources : GPUs for running models, CPUs for management services


• Networking and DNS : Routing traffic to the appropriate

... See more

• Query the RAG anyway and let the LLM itself chose whether to use the the RAG context or its built in knowledge


• Query the RAG but only provide the result to the LLM if it meets some level of relevancy (ie embedding distance) to the question


• Run the LLM both on it's own and with the RAG response, use a heuristic (or another LLM) to pick the best answer

... See more

Principles for growable tools

There are three critical pieces to building a tool that can grow around its users over time.

1. Design around play . Sometimes I call this design around experimentation . Using the tool for day-to-day work should involve playing and experimenting with what’s possible with the tool. Whether that’s writing small programs to

... See more