updated 1y ago
Just a moment...
- MLEs are happy to delegate experiment tracking and execution work to ML experiment execution frameworks, such as Weights & Biases 3 , but prefer to choose subsequent experiments themselves. To be able to make informed choices of subsequent experiments to run, MLEs must maintain awareness of what they have tried and what they haven’t (Lg2 calls ... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added
learnings from one experiment into the next, like a guided search to find the best idea (Lg2, Sm4,
Lg5). Lg5 described their ideological shift from random search to guided search:
Previously, I tried to do a lot of parallelization. If I focus on one idea, a week at a time,
then it boosts my productivity a lot more.
By following a guided search, engineers are, essentially, significantly pruning a large subset of
experiment ideas without executing them. While it may seem like there are unlimited computational
resources, the search space is much larger, and developer time and energy is limited. At the end of
the day, experiments are human-validated and deployed. Mature ML engineers know their personal
tradeoff between parallelizing disjoint experiment ideas and pipelining ideas that build on top of
each other, ultimately yielding successful deployments
- model-agnostic meta-learning, or MAML, it trains a model using two machine-learning processes, one nested inside the other.[...] setting his AI the challenge of learning more than one task led it to come up with its own version of a sparse model that outperformed human-designed ones.
from AI is learning how to create itself by Will Douglas Heaven
Kasper Jordaens added
- We found the ML engineering workflow to revolve around the following stages (Figure 1): (1) Data Preparation , which includes scheduled data acquisition, cleaning, labeling, and trans-formation, (2) Experimentation , which includes both data-driven and model-driven changes to increase overall ML performance, and is typically measured by metrics suc... See more
from "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. by Shreya Shankar
Nicolay Gerold added