First of all, I'd say you have a bigger problem where your company is trying to find nails with a hammer. That is where your sentiment comes from, and could be an obstacle for both you and the company. It's the same deal when I see people keep on talking about RAG, and nowadays "modular RAG", when really, you could treat everything as a software component, and just design a good software first that may or may not use LLMs.

Back to the original problem, it's not LLM vs. non-LLMs, it's when to use them. I have by this point co-developed 3 different applications, sometimes dabbled more into software side of things than not. Only one of them is chat-based, and that's the one I put the least effort on. The one I've put the most effort on, information retrieval is just part of the problem.

LLM usage is abundance, and one of the common use is to bootstrap NLP tasks where you can't have a model now. As you already know, you can reframe LLMs into other NLP tasks, but you also already pointed out that if you have a well trained model, LLMs shouldn't be needed. Very much agree so. But a lot of times, you don't have the training data. Training data means:

You already have a software/system that collects data

You already have a large enough dataset

You may or may not already have a data pipeline to transform raw data into your needs

You have time and budget for experimentation

You have well-defined target that doesn't easily change

And more...

In reality, a lot of times those things are not met. For example, one of our applications, we need to:

Extract information from text, based on "dynamically generated" or "user-specified" entities. That basically means zero-shot NER

Based on extracted entities, further "normalize them" into certain "dynamically generated" or "user-specified" classes, this again is a zero-shot text classification

Some information may live on images rather than text, again without previously known entity type

This type of IE can't just easily have a "well-trained BERT". Now, if there are common entities across clients, then you possibly can train an actual encoder model for them, but then you still need to actually analyze if that's a priority since APIs are getting cheaper everyday.

Another scenario is, if you're working on a greenfield project where you don't have enough data yet, you can still use LLMs to bootstrap, until you collect enough data, train your model. The bootstrap pattern actually helps software development to be very fast. As MLE, I think it's a missed opportunity to not consider LLMs at all, but it's also lazy to just slap LLMs on everything. One needs to consider, for every component that uses LLMs, can they be further broken down? Is it even necessary to use LLMs? If this component is bootstrapped now, can we design the software to ensure we have potential to train internal models in the future to optimize for cost as we scale?

From software development side, while working through codebases (I've learned TypeScript as a result :D), I also saw patterns that are making the codebase more maintainable. Where there used to be hardcoding, you can selectively use simple prompt to make the component more flexible. For example, your B2B application may have a lot of custom contracts, which still determine which features you may have. To not violate access of control, you can keep the feature access with programs themselves, but you can then have LLMs to analyze the custom contract and output "reasons" in the contract why certain features are not available. Why use LLMs? Because you don't need to hardcode the text and decision to analyze and upsell, and you don't need to add more code when there are new features or types of contracts.