Jev + LLM Cascade: Use Jev Before an Expensive Model
Send bounded decisions to Jev and reserve generation for the handler that needs it.
On this page
Separate judgment from generationA practical route mapUse a fixed dispatch tableAccount for the extra callHandle failures between stagesEvaluate the full outcomeSeparate judgment from generation
The cascade asks Jev who should handle a task. Deterministic code handles exact calculation, a fast LLM handles routine prose, a deeper model handles difficult synthesis, and review catches uncertainty or missing context. Jev does not generate the final explanation.
A practical route map
| Route | Owner | Example |
|---|---|---|
| simple | Code | Sum verified line items |
| normal | Fast LLM | Rephrase a short reply |
| complex | Deeper LLM | Synthesize several reports |
| uncertain | Human or clarification | Missing facts or conflicting instructions |
Use a fixed dispatch table
Store provider model IDs in configuration and map allowed labels to them. Check input sensitivity, data-handling requirements, and permissions before sending context to a downstream provider. The router should not decide where private data is allowed to go.
Account for the extra call
Routing adds latency and cost. Measure the total pipeline rather than only the generation bill. For an already trivial generation request, the classification step may not pay for itself. Cache only where your privacy and invalidation rules permit.
Handle failures between stages
A router timeout should select a documented fallback. A generation timeout is a separate failure with its own retry budget. Avoid repeated execution when a retry might duplicate a side effect. Store a request ID across stages to trace the final answer back to its route.
Evaluate the full outcome
Measure final answer quality, routing mistakes, end-to-end latency, and total cost. Include borderline tasks; a shorter prompt can still be difficult. Use the model routing tutorial for a complete local fixture and live decision program.