LLMs

TorchMultimodal (Beta Release)

Introduction

TorchMultimodal is a PyTorch library for training state-of-the-art multimodal multi-task models at scale. It provides:

• A repository of modular and composable building blocks (models, fusion layers, loss functions, datasets and utilities).


• A repository of examples that show how to combine these building

... See more

MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec. Watch a quick video introducing the project here.

• Multi-model serving, letting users run multiple models within the same process.


• Ability to run inference in parallel for vertical

... See more

A core research interest of mine is imagining new kinds of interfaces to text documents that are made possible by modern AI and software. I think an interesting place to look for such ideas may be interface designs for reading and writing legal documents .

Legal document-wrangling tools have a handful of properties that make it fertile ground for... See more

• Query the RAG anyway and let the LLM itself chose whether to use the the RAG context or its built in knowledge


• Query the RAG but only provide the result to the LLM if it meets some level of relevancy (ie embedding distance) to the question


• Run the LLM both on it's own and with the RAG response, use a heuristic (or another LLM) to pick the best answer

... See more

You can think your way into solving a deterministic system, but you cannot think your way into solving a probabilistic system.

The first thing that I want to call out is that deterministic software has edge cases, while probabilistic software has long tails.

I find that a lot of junior folks try to really think hard about edge cases around... See more

Setting up the necessary machine learning infrastructure to run these big models is another challenge. We need a dedicated model server for running model inference (using frameworks like Triton oder vLLM), powerful GPUs to run everything robustly, and configurability in our servers to make sure they're high throughput and low latency. Tuning the... See more

We generally lean towards picking more advanced commercial LLMs to quickly validate our ideas and obtain early feedback from users. Although they may be expensive, the general idea is that if problems can't be adequately solved with state-of-the-art foundational models like GPT-4, then more often than not, those problems may not be addressable... See more

When it comes to identifying where generative AI can make an impact, we dig into challenges that commonly:

• Involve analysis, interpretation, or review of unstructured content (e.g. text) at scale


• Require massive scaling that may be otherwise prohibitive due to limited resources


• Would be challenging for rules-based or traditional ML approaches

Some tasks are easier to improve via system design. While LLMs appear to follow remarkable scaling laws that predictably yield better results with more compute, in many applications, scaling offers lower returns-vs-cost than building a compound system. For example, suppose that the current best LLM can solve coding contest problems 30% of the

... See more