Let the Keywords Guess, Let the LLM Decide

Why we let keyword hints guess but never decide and the fallback chain that saves our AI agents routing when the LLM call fails.

A user types "send that to Sarah" into the agent. Somewhere in the routing logic, a keyword match fires on "send" and points straight at the email app. Feels right. Except Sarah wanted a calendar invite, not an email, and the agent just guessed wrong with total confidence.

This is the trap of keyword matching. It's fast. It's free. It's also often wrong, and wrong in a way that looks correct until it isn't.

I built our app-intent router, composio_router.py, around one hard rule: keywords never make the final call. They only guess. The LLM always decides. Let the keywords guess, let the LLM decide — that split is the whole lesson here.

Why Keyword Matching Alone Falls Apart

Keyword matching is tempting because it's instant and it costs nothing. Scan the message, check for words like "email," "sheet," or "meeting," and route based on a match. No API call. No wait time.

Here's where it breaks:

• The word "meet" shows up in calendar requests. It also shows up in casual chat that has nothing to do with scheduling.
• The word "sheet" works the same way — it might mean a spreadsheet, or it might mean a cheat sheet.
• Real messages mix intents too. "Update the tracker and let Sarah know" touches two apps in one line. A keyword scan picks one word and runs with it. Often, it picks wrong.

Confidence is the real danger here. A keyword match doesn't know it might be wrong. It just fires. Everything downstream trusts it.

The Two-Tier Fix Hints First, Judgment Always

This is the part that changed how our router behaves. Keyword matching still runs. It just lost its authority.

Tier One Keyword Matching as a Free, Instant Hint

The router still scans the message for known app keywords first. This step stays cheap and instant, and it produces a short list of likely apps.

Think of it as a hint, not a decision. It's a nudge in a direction, not a verdict.

Tier Two The LLM Always Gets the Final Say

Here's the rule that matters most: the LLM always sees the full message, every time. Not just the flagged keywords. Not a filtered version. The whole thing, plus the hint list as extra context.

The LLM can agree with the hint. It can also override it completely if the message means something the keyword scan missed. That override path is the whole point of building it this way.

Skipping this step to save time is exactly the mistake I wanted to avoid. Fast doesn't mean trustworthy. Speed is a bad reason to skip judgment on something a user is counting on you to get right.

What This Looks Like in Practice

# simplified, illustrative flow — not a literal code excerpt
hints = keyword_scan(message)          # tier one: fast, free, just a guess
decision = llm_decide(message, hints)  # tier two: sees everything, decides
route_to(decision.app)

Notice that hints get passed in as context, not as a filter. The LLM never receives a narrowed version of the message. It always sees the whole thing.

The Fallback Chain When the LLM Call Fails

APIs fail. Networks drop. A model call can time out or error, and a router that hasn't planned for that will just crash or misfire at the worst moment.

Our fallback chain handles that gracefully, in order:

1. Try the LLM decision with the full message and the keyword hints.
2. If that call fails, fall back to the keyword hints alone and route to the best-guessed app from tier one.
3. If even the hints are empty or unclear, offer every connected app as an option instead of guessing wrong in silence.

That last step matters more than it sounds. A confident wrong guess is worse than an honest "not sure, pick one."

Why This Distinction Actually Matters

I've watched teams build routing logic that stops at tier one, because it's cheap and it usually works. "Usually works" is a rough standard, though — not when the failure mode is an email getting routed somewhere it shouldn't.

The lesson isn't "don't use keyword matching." It's "don't let a cheap heuristic make decisions it wasn't built to make." Keywords are great at narrowing options fast. They're bad at reading intent, context, or a message that mixes two requests into one line.

How to Apply This Pattern to Your Own Agent

1. List every keyword your matcher uses to trigger a route, and check how many show up in messages that have nothing to do with that app.
2. Change your matcher's output from a decision to a hint. Pass the full user message to your LLM step no matter what the keyword scan found.
3. Build a real fallback chain. Decide what happens if the LLM call fails, what happens if hints are also empty, and make sure the last resort is transparency, not a silent wrong guess.
4. Test it against messages designed to trip up keyword matching on purpose. Mixed-intent sentences and ambiguous single words are the best test cases you have.

Common Mistakes to Avoid

• Letting a keyword match skip the LLM step for "obvious" cases. Nothing is obvious enough to skip judgment on user intent.
• Filtering the message before the LLM sees it, based on which keywords matched. This throws away context the model needs to catch what the keyword scan missed.
• Treating an LLM API failure as a dead end instead of a fallback trigger. A missing decision should degrade gracefully, not fail loudly.

Frequently Asked Questions

Why not just skip keyword matching and always call the LLM? Keyword matching still adds value as an instant, free hint that gives the LLM helpful context. Removing it entirely doesn't fix anything — the fix is making sure it never overrides the LLM's judgment.

Doesn't sending the full message to the LLM every time cost more? Yes, slightly more than pure keyword matching alone. That cost buys you correct routing on ambiguous or mixed-intent messages, which is a fair trade for anything user-facing.

What if the keyword hints and the LLM decision disagree? The LLM wins, every time. Hints are context, not a vote that gets averaged against the model's read of the full message.

Is a fallback chain really necessary for something this small? Yes. Any external API call can fail, and a routing system with no fallback just breaks at that exact moment instead of degrading gracefully.

The Bottom Line

A cheap, fast heuristic earns a place in your system as a hint, never as the decision-maker. Let the keywords guess, let the LLM decide, and build a fallback chain for the moment the LLM call itself doesn't come through. That's the whole shape of a routing system you can actually trust.

Want more breakdowns like this one? Subscribe for weekly notes on how we build and test AI agents, mistakes included.

Suggested internal links: One API Key to Rule Every LLM, Auditing Every Permission Your AI Agent Actually Uses