r/MachineLearning - Reddit
reddit.com
r/MachineLearning - Reddit
The Mac Studio is an absolute monster for inferencing; but there are a couple of caveats.
- Its slower, pound for pound, than a 4090 when dealing with models the 4090 can fit in its VRAM. So a 13b model on the 4090 is almost twice as fast as it running on the M2.
- The M1 Ultra Mac Studio with 128GB costs far less ($3700 or so) and the inference speed is
r/LocalLLaMA - Reddit
Nicolay Gerold added
M3 max is objectively worse than the M2 for inference.
The M2 ultra has a higher max RAM size of 192 GB
The M1 ultra has 128 GB max ram.
When it comes to these ram numbers something like 2/3 of it is available for inference.
So I see no reason why not to make a general recommendation for the M1 ultra unless you have some reason you want to run q5_K_M 1... See more
The M2 ultra has a higher max RAM size of 192 GB
The M1 ultra has 128 GB max ram.
When it comes to these ram numbers something like 2/3 of it is available for inference.
So I see no reason why not to make a general recommendation for the M1 ultra unless you have some reason you want to run q5_K_M 1... See more
r/LocalLLaMA - Reddit
Nicolay Gerold added
However development time, and maintenance can offset these savings. Hiring skilled data scientists, machine learning engineers, and DevOps professionals can be expensive and time consuming. Using available resources for “reimplementing” solutions hinder innovation and lead to a lack of focus. Since You not longer work on improving your model or pro... See more
Understanding the Cost of Generative AI Models in Production
Nicolay Gerold added
First of all, I'd say you have a bigger problem where your company is trying to find nails with a hammer. That is where your sentiment comes from, and could be an obstacle for both you and the company. It's the same deal when I see people keep on talking about RAG, and nowadays "modular RAG", when really, you could treat everything as a software co... See more
r/MachineLearning - Reddit
Nicolay Gerold added
MLEs are happy to delegate experiment tracking and execution work to ML experiment execution frameworks, such as Weights & Biases 3 , but prefer to choose subsequent experiments themselves. To be able to make informed choices of subsequent experiments to run, MLEs must maintain awareness of what they have tried and what they haven’t (Lg2 calls ... See more
Shreya Shankar • "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning.
Nicolay Gerold added
learnings from one experiment into the next, like a guided search to find the best idea (Lg2, Sm4,
Lg5). Lg5 described their ideological shift from random search to guided search:
Previously, I tried to do a lot of parallelization. If I focus on one idea, a week at a time,
then it boosts my productivity a lot more.
By following a guided search, engineers are, essentially, significantly pruning a large subset of
experiment ideas without executing them. While it may seem like there are unlimited computational
resources, the search space is much larger, and developer time and energy is limited. At the end of
the day, experiments are human-validated and deployed. Mature ML engineers know their personal
tradeoff between parallelizing disjoint experiment ideas and pipelining ideas that build on top of
each other, ultimately yielding successful deployments
When we deliver a model we make sure we don't reach X seconds of latency in our API. Before even going into performance of LLMs for classification, I can tell you that with the current available tech they are just infeasible.
Reply
reply
LinuxSpinach
•
5h ago
^ this. And especially classification as a task, because businesses don’t want to pay llm buck... See more
Reply
reply
LinuxSpinach
•
5h ago
^ this. And especially classification as a task, because businesses don’t want to pay llm buck... See more
r/MachineLearning - Reddit
Nicolay Gerold added
We're doing NER on hundreds of millions of documents in a specialised niche. LLMs are terrible for this. Slow, expensive and horrifyingly inaccurate. Even with agents, pydantic parsing and the like. Supervised methods are the way to go. Hell, I'd take an old school rule based approach over LLMs for this.
- Traditional AI - The most secure, understandable, and performant. However, Good implementations of traditional AI require that we define the rules behind the system, which makes it unfeasible for many of the use cases that the other 2 techniques thrive on.
- Supervised Machine Learning- Middle of the road b/w AI and Deep Learning. Good when we have
Devansh • How to Pick between Traditional AI, Supervised Machine Learning, and Deep Learning [Thoughts]
Nicolay Gerold added
Where would I add generative AI? Generative AI has the ease of accessibility of traditional AI, where people think it is understandable, but it does not have that feature in itself. It also has the opaque and costly nature of DL. Many companies are at the moment rushing into developing things with generative AI without having any prior foundation in AI and any processes set up to manage it: data ops, devops, …
Traditional AI forces you to think about how something works, understand the system, and then define the rules for it. ML lets you use features and feature importance to shortcut some. Deep Learning allows you to brute force it. Generative AI allows you to brute force without any background in DL.
Setting up the necessary machine learning infrastructure to run these big models is another challenge. We need a dedicated model server for running model inference (using frameworks like Triton oder vLLM), powerful GPUs to run everything robustly, and configurability in our servers to make sure they're high throughput and low latency. Tuning the in... See more
Developing Rapidly with Generative AI
Nicolay Gerold added