What to collect
- An indexing and retrieval pipeline with reproducible setup
- A labelled query set covering answerable, ambiguous, and unanswerable questions
- A retrieval-versus-generation error analysis with one measured improvement
RETRIEVAL & EVALUATION
See whether a candidate can make RAG work
Evaluate retrieval and answer quality separately. A fluent answer should not hide a poor retrieval pipeline.
01 / THE TASK
A practical starting point.
Build a retrieval pipeline over a supplied handbook containing overlapping policies, two document versions, and deliberately unanswerable questions. Return supporting passages with each answer and explain how you handle stale or conflicting information.
Adapt this example to the role, agree allowed tools, and give candidates the same instructions. It is a starting template, not a validated selection test.
02 / THE RUBRIC
Agree what good looks like.
Calibrate these suggested weights with your interviewers before the first candidate. Record the evidence behind each rating.
| Criterion | Weight | Stronger evidence | Needs investigation |
|---|---|---|---|
| Retrieval quality | 30% | Evaluates relevant-passage retrieval against labelled examples. | Selects chunks using a few hand-picked queries. |
| Grounded answers | 30% | Citations support the answer; absent evidence triggers an explicit abstention. | Citations are present but do not justify the answer. |
| Evaluation and debugging | 25% | Distinguishes retrieval failures from generation failures. | Changes prompts without identifying the failing stage. |
| Operational choices | 15% | Explains document updates, access boundaries, and cost. | Ignores stale documents and deployment constraints. |
03 / ILLUSTRATIVE REPORT
From observation to follow-up.
Example only · not a real candidate resultObserved
The sample retrieves the correct policy on eight of ten labelled queries, but answers from an older version in one conflicting-document case.
Reviewer’s next step
Discuss version filtering and how the candidate would detect this regression. The ten-query example is illustrative, not a validated benchmark.
Ask about their own work.
- Which failure belongs to retrieval, and which to generation?
- How would you stop an outdated policy from overriding a current one?
- What would make you change chunking rather than the prompt?
04 / SEE THE PRODUCT
Bring the brief to your demo.
This LunaPrompts product walkthrough introduces the assessment workflow. Ask the team to demonstrate your exact task, chosen tools, and report requirements in the environment you plan to use.
The walkthrough shows the product. It is not a recording of the illustrative exercise above or an independent performance benchmark.
Read the detailed workflow guide ↗PUT IT TO WORK
Start with the role you’re hiring for.
Bring a job description, a few must-have skills, and your hiring timeline. We’ll help you shape the evaluation.
Talk to us ↗