AGENT ENGINEERING

AI agent assessment: test tools, state, and recovery

An agent that completes the happy path is only part of the job. Test what it does when a tool fails, context changes, or an action needs approval.

Build this assessment with us ↗Download the assessment brief ↓

01 / THE TASK

A practical starting point.

Build an agent that triages support tickets using a supplied policy and a mock order API. It may look up orders and draft responses, but refunds require approval. Include a ticket containing instructions to ignore the policy, a tool timeout, and a repeated request. Use synthetic data and mock actions throughout.

Adapt this example to the role, agree allowed tools, and give candidates the same instructions. It is a starting template, not a validated selection test.

What to collect

  • A runnable agent with explicit tool contracts and approval boundaries
  • A trace for normal, injected-instruction, timeout, and duplicate-request scenarios
  • A small evaluation set and a written explanation of state, retries, and stopping rules

02 / THE RUBRIC

Agree what good looks like.

Calibrate these suggested weights with your interviewers before the first candidate. Record the evidence behind each rating.

Suggested rubric · 100% total weight
CriterionWeightStronger evidenceNeeds investigation
Tool use and boundaries30%Validates tool arguments and requires approval before a mock refund.Treats a retrieved instruction as permission to perform an action.
State and recovery30%Handles timeouts and repeated requests without duplicate actions or unbounded retries.Loses task state or repeats an action after a retry.
Evaluation and traces25%Uses repeatable cases and traces to identify where behavior diverges from policy.Shows one successful run without checking failure scenarios.
Engineering judgment15%Explains when a deterministic step is preferable and defines stopping conditions.Adds more agent steps without explaining their cost or purpose.

03 / ILLUSTRATIVE REPORT

From observation to follow-up.

Example only · not a real candidate result

Observed

The illustrative agent requests approval correctly, but issues the same mock action twice when the first response times out.

Reviewer’s next step

Ask the candidate to add idempotency handling and demonstrate the retry case. A successful final message does not prove that intermediate actions were correct.

Ask about their own work.

  1. Where does trusted policy end and untrusted ticket content begin?
  2. How do you distinguish a failed action from a successful action whose response was lost?
  3. What would make this workflow stop and ask a human for help?
Explore submission-specific AI Viva ↗

Customize this scoring rubric ↗

04 / SEE THE PRODUCT

Bring the brief to your demo.

This LunaPrompts product walkthrough introduces the assessment workflow. Ask the team to demonstrate your exact task, chosen tools, and report requirements in the environment you plan to use.

The walkthrough shows the product. It is not a recording of the illustrative exercise above or an independent performance benchmark.

Read the detailed workflow guide ↗

PUT IT TO WORK

Start with the role you’re hiring for.

Bring a job description, a few must-have skills, and your hiring timeline. We’ll help you shape the evaluation.

Talk to us ↗