Artificial Intelligence1 min read

Not Everything Needs an LLM

By · Published by Everything Blog

In short

PostHog has demonstrated that decision models can be up to 444.6 times cheaper and 193.6 times faster than frontier models for simple tasks. In its Replay Vision product, the company utilizes targeted yes-or-no queries to stop 98.7 percent of clean session summaries from reaching Gemini. Teams adjust probability thresholds depending on whether missing sensitive data poses a risk or unwarranted interruptions must be prevented. To handle shifting calibration requirements across different model updates, the company built an internal AI gateway to benchmark models against hand-labeled data.

Key points

  • Decision models provide structured typed answers alongside explicit probability scores to…: Decision models provide structured typed answers alongside explicit probability scores to simplify uncertainty tracking.
  • Engineering teams can deploy decision models as cost-effective routers to assign simple t…: Engineering teams can deploy decision models as cost-effective routers to assign simple tasks like lookups to cheaper options such as GPT Luna.
  • Calibrations shift across versions and tasks, requiring internal AI gateways to continual…: Calibrations shift across versions and tasks, requiring internal AI gateways to continually test system performance against hand-labeled datasets.

Sources checked by our analysis pipeline: 1

LLMs love to yap. Ask a simple question and you get a few hundred words of "it depends." It's annoying as hell.

This is why we love decision models like Jev for simpler tasks. You ask a structured question and get a typed answer with a probability, which looks something like this:

Noul. Are hedgehogs mammals? → Yes, 0.99

Choice. Which animal family do hedgehogs belong to? → Mammal, 0.99

Score. What is the current conservation status of hedgehogs? → Least Concern, 0.77

Whereas LLMs return free-form and overly confident prose, decision models make model uncertainty explicit. In a calibrated model, answers that score 0.8 are right about 80% of the time. That means a score of 0.5 is a coin flip.

We've been experimenting with them at PostHog and several use cases have made it to production. This is everything we've learned about using decision models so far: when to use them, how to use them, and mistakes to avoid.

1. Use them to gate expensive LLM calls

Decision models are up to 444.6x cheaper and 193.6x faster than…

Artificial Intelligence