# See whether a candidate can make RAG work

https://lunaprompts.com/solutions/rag-assessment

Illustrative assessment template. Adapt and calibrate for the role before use.

Evaluate retrieval and answer quality separately. A fluent answer should not hide a poor retrieval pipeline.

## Task

Build a retrieval pipeline over a supplied handbook containing overlapping policies, two document versions, and deliberately unanswerable questions. Return supporting passages with each answer and explain how you handle stale or conflicting information.

## Deliverables

- An indexing and retrieval pipeline with reproducible setup
- A labelled query set covering answerable, ambiguous, and unanswerable questions
- A retrieval-versus-generation error analysis with one measured improvement

## Suggested rubric

### Retrieval quality — 30%
Stronger evidence: Evaluates relevant-passage retrieval against labelled examples.
Investigate: Selects chunks using a few hand-picked queries.

### Grounded answers — 30%
Stronger evidence: Citations support the answer; absent evidence triggers an explicit abstention.
Investigate: Citations are present but do not justify the answer.

### Evaluation and debugging — 25%
Stronger evidence: Distinguishes retrieval failures from generation failures.
Investigate: Changes prompts without identifying the failing stage.

### Operational choices — 15%
Stronger evidence: Explains document updates, access boundaries, and cost.
Investigate: Ignores stale documents and deployment constraints.

## Illustrative report (not a real candidate result)

The sample retrieves the correct policy on eight of ten labelled queries, but answers from an older version in one conflicting-document case.

Discuss version filtering and how the candidate would detect this regression. The ten-query example is illustrative, not a validated benchmark.

## Follow-up questions

- Which failure belongs to retrieval, and which to generation?
- How would you stop an outdated policy from overriding a current one?
- What would make you change chunking rather than the prompt?

Discuss your requirements: https://lunaprompts.com/pricing