"We Have No Idea How Models will Behave in Production until ...

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning.

How ML engineers operationalize machine learning, their workflow stages (data preparation, experimentation, evaluation and deployment, monitoring and response), and the challenges they face in each stage.

arxiv.org

RelatedHighlights

So I've been doing consulting for a couple of rad companies and the playbook usually is the same thing. The first thing you do is you joined the team and you'd realize the engineers aren't really being quantitative with how they're describing problems. They're using words like "it looks like", "it feels like", "it look... See more

jason liu x.com

Thumbnail of www-x-com-dexhorthy-status-1913246092462104671-0282be2e88284c4d

may have hit a nerve here...it all started with trying to understand how production AI systems actually work FOLKS - I've tried every agent framework out there and talked to many strong founders building impressive things with AI BUT I was surprised to find that most successful AI systems ar... See more

dex

x.com

a quick note from @karpathy talk, which afterwards I was fully immersed in the talk I stopped writing down/taking notes: While it is interesting to think about LLMs in terms of "electricity" -> CAPEX to train an LLM (~= to build the grid) -> OPEX to serve intelligence over increasingly homogeneous API (prompt, image, to... See more

ADAM

x.com