Introduction: A Counterintuitive Decision
In a market where everyone seems to be developing language model routers (LLM), why would a company like Manifest choose to backtrack and deprecate its own? This decision, which might seem surprising at first glance, is actually based on practical insights and real-world feedback.
The LLM Router Phenomenon
LLM routers have exploded onto the market with the promise of reducing inference costs by dynamically selecting the most appropriate model for each request. In theory, this seems like the perfect solution: why engage an expensive model for a simple task? However, practice shows that this approach has significant limitations.
A Complex Context
When Manifest launched its LLM router in March, the goal was clear: optimize costs and improve efficiency. The router classified each request into four levels of complexity: simple, standard, complex, and reasoning. But after just four months and over 7000 users, the results were mixed. The main issue? Task complexity cannot be deduced from the prompt alone.
The Limits of LLM Routers
Complexity Beyond the Prompt
The prompt itself is just the trigger. Much of the context that determines task complexity is discovered later through tool calls, web searches, etc. For example, "evaluate the tests for the repo $GIT_REPO and improve them" can be simple or complex depending on the context, a factor the router could not anticipate.
Cost and Reliability Issues
Despite promises of cost reduction, using LLM routers led to reliability issues and unforeseen costs due to incorrect routing. Many users reported issues on GitHub, leading to lively discussions within the community.
Why a Single Model Can Be More Effective
Ultimately, Manifest concluded that, for most use cases, sticking to a proven and robust model was the best option. This reduces the risk of errors and simplifies development. Moreover, it allows for better cost control without compromising reliability.
The Importance of Stability and Control
For developers and businesses, stability and control are crucial. A single model simplifies the infrastructure and reduces system complexity, which is often more beneficial than the potential cost savings promised by LLM routers.
Conclusion
Manifest's decision to deprecate its LLM router highlights the importance of efficiency and reliability in AI development. As the market continues to evolve, it's essential to focus on solutions that offer real added value.
Let's discuss your project in 15 minutes.