Announcing Together Inference Engine – the fastest inference available

TL;DR

We are thrilled to announce the release of the FASTEST Voice LLM to date! Experience real-time speech streaming from text in 300ms or less. Dive in and test it using our Playground, available SDKs, or these Replit demos for both Nodejs and Python and a chatGPT integration.

Introduction

At PlayHT, our vision revolves around redefining human int... See more

One major challenge is the inference time of the models. While models like ChatGPT have improved in speed, they still take time to process information. When dealing with a large number of agents, there can be significant latency in real-time interactions. Optimizations and fine-tuning will be necessary to make the models faster and more efficient. ... See more

Setting up the necessary machine learning infrastructure to run these big models is another challenge. We need a dedicated model server for running model inference (using frameworks like Triton oder vLLM), powerful GPUs to run everything robustly, and configurability in our servers to make sure they're high throughput and low latency. Tuning the in... See more