"We Have No Idea How Models will Behave in Production until ...

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning.

How ML engineers operationalize machine learning, their workflow stages (data preparation, experimentation, evaluation and deployment, monitoring and response), and the challenges they face in each stage.

arxiv.org

updated 5mo ago

they could try “switching to a different model, augmenting the training data in some way, collecting more or different kinds of data, post-processing outputs, changing the objective function, or something else.” Our interviewees recommended focusing on experiments that provided additional context to the model, typically via new features, to get the... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
Over time, ML pipelines may turn into “jungles” of rules and models . Sculley et al . [87] introduce the phrase “pipeline jungles” (i.e., different versions of data transformations and models glued together), which was later adopted by participants in our study. While prior work demonstrates their existence and maintenance challenges, we provide in... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
business is open. [It might say] “9 am,” but the model doesn’t know that. So if we detect
time, then we filter that [reply]. We have a lot of filters.
MLEs are happy to delegate experiment tracking and execution work to ML experiment execution frameworks, such as Weights & Biases 3 , but prefer to choose subsequent experiments themselves. To be able to make informed choices of subsequent experiments to run, MLEs must maintain awareness of what they have tried and what they haven’t (Lg2 calls ... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
learnings from one experiment into the next, like a guided search to find the best idea (Lg2, Sm4,
Lg5). Lg5 described their ideological shift from random search to guided search:
Previously, I tried to do a lot of parallelization. If I focus on one idea, a week at a time,
then it boosts my productivity a lot more.
By following a guided search, engineers are, essentially, significantly pruning a large subset of
experiment ideas without executing them. While it may seem like there are unlimited computational
resources, the search space is much larger, and developer time and energy is limited. At the end of
the day, experiments are human-validated and deployed. Mature ML engineers know their personal
tradeoff between parallelizing disjoint experiment ideas and pipelining ideas that build on top of
each other, ultimately yielding successful deployments
“I look for features from data scientists, [who have ideas of] things that are correlated with what I’m trying to predict.” We found that organizations explicitly prioritized cross-team collaboration as part of their ML culture. Md3 said: We really think it’s important to bridge that gap between what’s often, you know, a [subject matter expert] in ... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
We found the ML engineering workflow to revolve around the following stages (Figure 1): (1) Data Preparation , which includes scheduled data acquisition, cleaning, labeling, and trans-formation, (2) Experimentation , which includes both data-driven and model-driven changes to increase overall ML performance, and is typically measured by metrics suc... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
(1) Tasks (a) Data collection, cleaning & labeling: human annotators , exploratory data analysis (b) Embeddings & feature engineering: normalization , bucketing / binning , word2vec (c) Data modeling & experimentation: accuracy , F1-score , precision , recall (d) Testing: scenario testing , AB testing , adaptive test-data (2) Biz/Org Ma... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
Amershi et al . [3] state that software teams “flight” changes or updates to ML models, often by testing them on a few cases prior to live deployment. Our work provides further context into the evaluation and deployment process for production ML pipelines: we found that several organizations, particularly those with many customers, employed a multi... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
engineers continuously monitored features for and predictions from production models (Lg1, Md1, Lg3, Sm3, Md4, Sm6, Md6, Lg5, Lg6): Md1 discussed hard constraints for feature columns (e.g., bounds on values), Lg3 talked about monitoring completeness (i.e., fraction of non-null values) for features, Sm6 mentioned embedding their pipelines with "comm... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago
Several engineers also maintained fallback models for reverting to: either older or simpler versions (Lg2, Lg3, Md6, Lg5, Lg6). Lg5 mentioned that it was important to always keep some model up and running, even if they “switched to a less economic model and had to just cut the losses.” Similarly, when doing data science work, both Passi and Jackson... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added 5mo ago

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar

from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar