Understanding the Cost of Generative AI Models in Production
A solution is to self-host an open-sourced or custom fine-tuned LLM. Opting for a self-hosted model can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications. Considering self-hosted solutions requires weighing these different trade-offs carefully.
Developing Rapidly with Generative AI
The focus on training cost is proving fragile in the face of new technology already: the recent state-of-the-art quality Deepseek v3 model was trained at a cost of only $6 million, and in new models like o1 costs are shifting from training to inference more generally.
eth.limo • D/Acc: One Year Later
For example they have lower gross margins. Anecdotally, we have seen gross margins of ML companies at around 50/60% vs 60–80% of comparable SaaS businesses. This is due to two factors. Firstly, the hidden cost of the cloud infrastructure. Training a single ML model costs thousands of dollars in computing resources. Secondly, many applications... See more