LLM-as-Judge Scoring Rubric

You are an expert evaluator assessing the quality of an AI-generated response.

Evaluate the response below using this rubric. For each dimension, give a score from 1 to 5 and a one-sentence justification.

Scoring rubric:
- Accuracy (1–5): Is the response factually correct based on the provided context?
- Groundedness (1–5): Does the response cite or reference the provided context? Does it avoid adding information not in the context?
- Relevance (1–5): Does the response directly answer the question asked?
- Completeness (1–5): Does the response cover all important aspects of the question?

Scoring guide: 1 = Very poor, 2 = Poor, 3 = Acceptable, 4 = Good, 5 = Excellent

---
Question: {{QUESTION}}

Context provided to the AI: {{RETRIEVED_CONTEXT}}

AI response to evaluate: {{AI_RESPONSE}}
---

Output your evaluation in this exact format:
Accuracy: [score]/5 — [one sentence justification]
Groundedness: [score]/5 — [one sentence justification]
Relevance: [score]/5 — [one sentence justification]
Completeness: [score]/5 — [one sentence justification]
Overall: [average score]/5

What this prompt does

Implements automated evaluation of LLM outputs using a structured scoring rubric. Returns a score (1–5) per dimension plus a one-line justification for each score, enabling scalable quality assessment.

When to use it

• Evaluating RAG pipeline output quality at scale
• A/B testing two prompt versions on the same dataset
• Monitoring production LLM responses for quality regressions

When NOT to use it

• For highly subjective creative tasks — rubric criteria must be objectively verifiable
• When you cannot afford the cost of a second LLM call per evaluation

The Prompt

Why it's structured this way

Separate dimensions — A single "quality" score conflates orthogonal failure modes. Separate scores pinpoint which part of your pipeline is failing.

"One sentence justification" — Forces the model to commit to a specific reason. Without this constraint, justifications become vague and lose diagnostic value.

"Exact format" instruction — Enables regex or string parsing of scores programmatically.

Context included in eval prompt — The judge must see the same context the original model had to evaluate groundedness accurately.

Variations