OpenAI Platform

• Lower max tokens : for requests with a similar token generation count, those that have a lower max_tokens parameter incur less latency.


• Include stop sequences : to prevent generating unneeded tokens, add a stop sequence. For example, you can use stop sequence

Intuition : Prompt tokens add very little latency to completion calls. Time to generate completion tokens is much longer, as tokens are generated one at a time. Longer generation lengths will accumulate latency due to generation required for each token.