A solution is to self-host an open-sourced or custom fine-tuned LLM. Opting for a self-hosted model can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications. Considering self-hosted solutions requires weighing these different trade-offs carefully.

The focus on training cost is proving fragile in the face of new technology already: the recent state-of-the-art quality Deepseek v3 model was trained at a cost of only $6 million, and in new models like o1 costs are shifting from training to inference more generally.

For example they have lower gross margins. Anecdotally, we have seen gross margins of ML companies at around 50/60% vs 60–80% of comparable SaaS businesses. This is due to two factors. Firstly, the hidden cost of the cloud infrastructure. Training a single ML model costs thousands of dollars in computing resources. Secondly, many applications... See more