Check AI Citations with Jev

What you’ll build

Classify whether a quoted source supports, contradicts, or leaves a claim unresolved, then apply a review gate. The result is a runnable decision pipeline, with a fixture mode for checking local behavior and a live mode for evaluating your own TypeSafe account. It prints a recommendation without performing external side effects.

Architecture

Claim + quoted passage + context + source metadata → support / contradict / insufficient → accept / review / reject

Evidence matching is narrower than fact-checking. A source can support a claim and still be stale, fabricated, or wrong. Retrieve the source independently, preserve its metadata, and distinguish the support judgment from source authenticity.

Prerequisites and sample input

Use Python 3.10+; these standalone examples use only the standard library. Live evaluation also requires TYPESAFE_API_KEY in the environment. Download the complete script below rather than copying disconnected fragments.

{
  "claim": "Customers can cancel their subscription at any time.",
  "quoted_passage": "You may cancel your subscription at any time from Billing.",
  "surrounding_context": "Cancellation stops future renewal; refund rules are separate.",
  "source_metadata": {
    "id": "policy-1",
    "title": "Subscription policy",
    "retrieved_at": "2026-09-20"
  }
}

Step 1 — Prepare the evidence

Keep the input shape stable and distinguish verified application facts from user claims. Evidence matching is narrower than fact-checking. A source can support a claim and still be stale, fabricated, or wrong. Retrieve the source independently, preserve its metadata, and distinguish the support judgment from source authenticity.

Step 2 — Define the atomic questions

{
  "verdict": {
    "type": "choice",
    "instructions": "Does the quoted passage support the claim in its surrounding context?",
    "criteria": {
      "supports": "The passage supports the claim as stated",
      "contradicts": "The passage conflicts with the claim",
      "insufficient": "The source does not establish the claim"
    }
  }
}

The question names map outputs back to your code. They are not inference instructions. Put the actual judgment in instructions, and use criteria for category descriptions or ordered levels.

Step 3 — Read answers and apply a gate

The standalone script checks required answer keys, types, allowed categories, numeric ranges, and confidence. Missing or malformed data stops the decision path. The following action policy uses educational thresholds; none have been measured on your data.

def decide(answers, state):
    verdict = answers["verdict"]
    if verdict["confidence"] < .9:
        route = "review"
    elif verdict["choice"] == "supports":
        route = "accept_evidence_match"
    elif verdict["choice"] == "contradicts":
        route = "reject"
    else:
        route = "seek_more_evidence"
    return {"route": route, "source_id": state["source_metadata"]["id"],
            "source_authenticity_verified": False}

Step 4 — Keep execution separate

The program prints a route or candidate result. A real executor must apply its own permissions, validation, idempotency, and confirmation requirements. A model label is evidence for a decision, not authorization to perform a consequential action.

Run it

Download the complete citation-checking.py program. Then run:

python citation-checking.py
# After configuring TYPESAFE_API_KEY, opt in to a live billed call:
python citation-checking.py --live

Fixture mode makes no network request and requires no credential. Live mode makes a single call with a 30-second timeout. It does not silently retry or execute any downstream action. For a production queue, add a bounded retry policy for transient failures and a durable review destination.

Expected fixture result

{
  "route": "accept_evidence_match",
  "source_id": "policy-1",
  "source_authenticity_verified": false
}

This output is deterministic fixture data, not a measured Jev response. A live model may produce different values and routes. Keep the full returned probability distributions when diagnosing that difference.

Read the probabilities

A Choice winner alone does not reveal ambiguity. Compare its confidence and distribution with the selected label. A Score is an ordered semantic value; use its legend before applying numeric thresholds. A Noul is the probability of yes and has no separate confidence field.

Failure cases and production improvements

A quote can omit a negation or a nearby exception. Test dates, quantifiers such as all versus some, and claims that combine two propositions when the passage supports only one. Split compound claims before evaluating.

Pin the tested model, log the returned version and rubric revision, and keep a small evaluation set under version control. Re-test after changes to inputs, provider, questions, or policy. Avoid logging sensitive input by default.

Complete code

The downloadable standalone script includes request construction, fixture data, response validation, the decision function, and command-line execution. It uses the direct HTTP API so no SDK dependency is required. For SDK versions of the core ticket request, see Python and JavaScript.

What to test before shipping

• A matching passage is accepted as an evidence match, not universal truth.
• A contradiction is rejected.
• Low confidence requires review.
• Insufficient evidence is never interpreted as supports.
• Network failures, invalid JSON, and missing fields must never become an automatic action.
• Compare several thresholds on labeled data and record the resulting review volume.