How to Test LLM Agents Before Launch

Testing an LLM agent before launch is different from testing a standard API endpoint. You are testing language, reasoning, tool use, state, retrieval, permissions, latency, cost, and failure behavior at the same time.

A strong pre-launch test plan should answer five practical questions:

• Does the agent understand the user’s goal?
• Does it choose the right tools and call them correctly?
• Does it stay within policy, permissions, and business rules?
• Does it recover cleanly when tools, retrieval, or model calls fail?
• Can your team debug the full run when something goes wrong?

If you cannot answer those questions with evidence, the agent is not ready for production traffic.

Start by defining what the agent is allowed to do

Before you write evals, define the agent’s operating boundaries. This sounds basic, but many agent failures come from vague product behavior rather than model weakness.

Write down the agent’s scope in plain language:

• Supported tasks: For example, “create a support ticket,” “summarize a customer account,” or “draft a renewal email.”
• Unsupported tasks: For example, “issue refunds over $500,” “change billing ownership,” or “provide legal advice.”
• Allowed tools: List each tool, what it does, required inputs, and expected outputs.
• Permission rules: Define what the agent can access based on user role, workspace, account status, or region.
• Escalation rules: Define when the agent should ask for clarification, refuse, or hand off to a human operator.

This spec becomes the baseline for your test cases. If the agent is dynamic and can decide which steps to take at runtime, read more about dynamic agents and make sure your tests cover multiple valid paths, not one fixed sequence.

Build a test set from real user behavior

A useful test set should include more than happy paths. Pull examples from support tickets, product analytics, sales calls, internal dogfooding, and beta users. If you do not have production data yet, write realistic synthetic cases based on expected workflows.

Start with 50 to 100 test cases before launch. That is usually enough to reveal major issues without slowing the team down. For higher-risk agents, such as finance, healthcare, security, or customer-facing agents with write access, aim for 200 or more cases before launch.

Include these categories

• Happy paths: Clear requests the agent should complete successfully.
• Ambiguous requests: Prompts missing required details, such as “move my meeting” without a date or attendee.
• Out-of-scope requests: Tasks the agent should refuse or redirect.
• Permission edge cases: Users asking for data they should not access.
• Tool failure cases: Timeouts, malformed responses, empty results, or 500 errors.
• Retrieval failures: Missing documents, irrelevant chunks, stale data, or conflicting sources.
• Multi-step tasks: Requests that require planning, tool use, validation, and final response generation.
• Adversarial prompts: Attempts to bypass policy, reveal hidden instructions, or force unsafe actions.

For each test case, store the input, expected behavior, required tool calls, disallowed actions, and pass criteria. Avoid vague labels like “good answer.” Use specific checks such as “must ask for the customer ID before calling the billing tool” or “must not call delete_user without admin approval.”

Test the prompt, tools, and orchestration separately

Agent failures often look like prompt failures, but the root cause may sit in a tool schema, retrieval step, routing rule, or state update. Test each layer on its own before you run full end-to-end evals.

Prompt behavior

Test whether the model follows the system prompt, handles uncertainty, and formats responses correctly. Include cases where the right answer is to ask a follow-up question.

Example checks:

• The agent explains next steps in fewer than 120 words.
• The agent does not invent account data when retrieval returns no result.
• The agent refuses requests for private data from another workspace.
• The agent asks for missing required fields before taking action.

Tool schemas

Test every tool with valid inputs, invalid inputs, missing fields, and unexpected outputs. If your agent uses OpenAI’s agent tooling, you can connect tracing and prompt management with the OpenAI Agents SDK integration.

For each tool, verify:

• The schema names match the prompt language.
• Required fields are actually required.
• Error messages are structured enough for the agent to recover.
• The tool cannot perform dangerous actions without server-side permission checks.
• The tool response includes stable identifiers, not only display names.

Orchestration and planning

If your agent decomposes work into steps, test the plan before testing the final answer. A plan-and-execute setup can fail even when each individual prompt looks good. For more background, see plan-and-execute agents.

Useful planning checks include:

• Does the agent create a reasonable sequence of actions?
• Does it skip unnecessary tools?
• Does it validate tool output before moving to the next step?
• Does it stop after the task is complete?
• Does it avoid repeated loops when a tool returns no result?

Run deterministic checks where you can

LLM outputs vary, but many agent behaviors can still be tested with deterministic checks. Use exact or structured assertions for anything that should not be subjective.

Good deterministic checks include:

• Tool name called or not called.
• Required JSON fields present.
• Number of tool calls under a limit, such as fewer than 6 calls per run.
• No call to restricted tools for non-admin users.
• Final answer includes a ticket ID after ticket creation.
• Latency under a target, such as p95 below 8 seconds.
• Cost under a budget, such as less than $0.03 per successful run.

Use model-graded evaluations for qualities that require judgment, such as tone, helpfulness, and reasoning quality. Keep those rubrics specific. A vague evaluator prompt will produce noisy results.

Example evaluation rubric

• Task success: Did the agent complete the requested task or ask for the correct missing information?
• Tool correctness: Did the agent choose the right tool and pass valid arguments?
• Policy compliance: Did the agent avoid restricted actions and unsafe disclosures?
• Grounding: Did the agent base its response on retrieved or tool-provided data?
• User experience: Was the answer concise, clear, and actionable?

Score each area from 1 to 5. Set a launch bar before testing starts. For example, require at least 95% pass rate on permission checks, 90% pass rate on task success, and zero critical safety failures.

Trace every agent run

You need traces before launch, not after the first incident. A useful trace records the full path of an agent run: user input, prompt version, model parameters, retrieval results, tool calls, tool outputs, intermediate reasoning summaries, final response, latency, cost, and errors.

Without traces, you will waste time guessing. With traces, you can inspect the exact point where the agent failed. Maybe the prompt was fine, but retrieval returned the wrong document. Maybe the agent chose the right tool, but the tool returned an empty payload. Maybe a prompt change increased token usage by 40%.

At minimum, capture:

• Prompt template and prompt version.
• Model name and parameters.
• User input and relevant metadata.
• Retrieved documents or context chunks.
• Tool call names, arguments, outputs, and errors.
• Final answer.
• Token count, cost, and latency.
• Evaluation results tied to the run.

This data lets you compare prompt versions and reproduce failures during pre-launch review.

Test memory and state carefully

Agents that store or reuse state need extra testing. A stateless chatbot can give a poor answer. A stateful agent can carry forward the wrong assumption, use stale data, or expose information from a previous user session.

Test state behavior with cases like these:

• A user changes their mind halfway through a task.
• A user gives updated information that conflicts with earlier input.
• Two users from different workspaces ask similar questions.
• A session resumes after several hours or days.
• The agent retrieves old context that should no longer apply.

Verify that the agent stores only what it should store. Sensitive fields, such as API keys, passwords, payment details, and private customer data, should be redacted or excluded unless there is a clear product requirement and access control in place.

Run permission and security tests before beta

Do not wait until production to test permissions. LLM agents can make permission bugs easier to trigger because users can ask for restricted data in natural language.

Build a permission matrix with roles, resources, and allowed actions. Then test it directly.

Important rule: do not rely on the prompt as your only security layer. Enforce permissions in your backend tools. The agent can decide what to request, but your system should decide what is allowed.

Test retrieval quality if the agent uses RAG

If your agent retrieves context, evaluate retrieval separately from generation. A good model cannot reliably answer from bad context.

Measure retrieval with a labeled set of questions and expected source documents. Track:

• Recall: Did retrieval include the needed document?
• Precision: Were the retrieved chunks actually relevant?
• Freshness: Did the agent use the latest version?
• Permission filtering: Did retrieval exclude documents the user cannot access?
• Context size: Did retrieval include too much irrelevant text?

For example, if the user asks, “What is our refund policy for annual plans in Germany?” your test should verify that retrieval includes the current refund policy, the regional billing terms, and no unrelated support macros.

Stress test loops, retries, and failure recovery

Agents need clear stopping behavior. Without it, a tool failure can turn into a retry loop that burns budget and creates a poor user experience.

Test these failure patterns:

• Tool timeout on the first attempt.
• Tool returns a validation error.
• Tool returns a partial result.
• Retrieval returns no documents.
• Model returns malformed JSON.
• External API rate limit is reached.

Set hard limits before launch. For example:

• Maximum 2 retries per failed tool call.
• Maximum 8 total tool calls per user request.
• Maximum 30 seconds total runtime for synchronous flows.
• Fallback response when the agent cannot complete the task.

The fallback should be useful. “Something went wrong” is rarely enough. A better response is, “I could not reach the billing system, so I could not update the payment method. Try again in a few minutes or contact support with account ID 1842.”

Compare static and dynamic agent behavior

Some agents follow a fixed workflow. Others choose tools and steps dynamically. Your testing strategy should match the architecture.

For fixed workflows, test every branch in the flow. These resemble static agents, where the system controls the sequence more tightly.

For dynamic agents, test outcome quality and guardrails across many possible paths. You may not know the exact sequence ahead of time, so you need assertions around allowed tools, required validations, and final task success.

If you are using a compiler-style approach to plan and optimize multi-step LLM workflows, review the concept of an LLM compiler and test both the generated plan and the executed run.

Use versioned prompts and regression tests

Prompt changes can fix one case and break ten others. Treat prompts like application code. Version them, review changes, and run regression tests before release.

A practical workflow looks like this:

1. Create or update a prompt version.
2. Run the full eval suite against the new version.
3. Compare pass rate, cost, latency, and failure categories against the current production version.
4. Inspect failed traces and update the prompt, tools, or test expectations.
5. Approve the prompt only when it meets the launch bar.

Keep a stable holdout set that you do not tune against every day. This helps catch overfitting, where the agent performs well on your known tests but poorly on new user requests.

Run human review on high-risk cases

Automated evals are necessary, but they will not catch every product issue. Add manual review for high-risk categories, especially before launch.

Ask reviewers to inspect:

• Permission refusals.
• High-cost runs.
• Long multi-step runs.
• Cases where evaluator scores disagree.
• Customer-facing responses for sensitive topics.
• Runs that call write-action tools, such as update, delete, send, refund, or approve.

Reviewers should label the root cause, not only pass or fail. Useful labels include prompt issue, retrieval issue, tool schema issue, missing product rule, model limitation, permission bug, and flaky external service.

Set launch gates

Decide what “ready” means before the final week of launch. Your launch gate should include quality, reliability, cost, and safety criteria.

Example launch gates:

• At least 90% task success on the main eval set.
• 100% pass rate on critical permission tests.
• Zero unsafe write actions in adversarial tests.
• p95 latency under 10 seconds for synchronous requests.
• Average cost under the product budget, such as $0.02 per run.
• Trace coverage for 100% of beta runs.
• Fallback behavior verified for tool timeouts and retrieval misses.

If the agent misses one of these gates, do not hide it under an average score. A 95% overall pass rate can still include a critical failure, such as exposing another customer’s data.

Run a staged rollout

After pre-launch testing, release the agent in stages. Start with internal users, then a small beta group, then a limited production percentage.

A simple rollout plan:

1. Internal dogfood: 20 to 50 users, trace every run, review failures daily.
2. Private beta: 5% to 10% of target users, monitor task success and escalation rate.
3. Limited production: 25% rollout with alerts for cost, latency, errors, and policy failures.
4. Full release: Expand only after metrics stay stable for at least one full business cycle.

Keep rollback simple. You should be able to disable the agent, revert to a previous prompt version, or remove write-action tools without shipping new application code.

Pre-launch testing checklist

• Defined supported and unsupported tasks.
• Documented tool schemas and permission rules.
• Built a test set with happy paths, edge cases, and adversarial prompts.
• Tested prompts, tools, retrieval, and orchestration separately.
• Added deterministic assertions for tool use and policy rules.
• Added model-graded evals with clear rubrics.
• Traced every agent run with prompt version, tool calls, cost, latency, and outputs.
• Tested memory, state, and session boundaries.
• Stress tested retries, timeouts, malformed outputs, and no-result cases.
• Compared the new prompt version against the current baseline.
• Completed manual review for high-risk cases.
• Set launch gates and rollback paths.

Final thoughts

Testing LLM agents before launch is about reducing uncertainty. You will never test every possible user request, but you can test the behaviors that matter most: tool correctness, permissions, grounding, recovery, latency, cost, and traceability.

The strongest teams treat agent testing as an ongoing release process. Every prompt change, tool update, retrieval change, and model upgrade should run through the same evaluation pipeline. That discipline makes agent behavior easier to debug and safer to improve.

PromptLayer helps AI teams manage prompts, run evaluations, trace agent behavior, and compare versions before production issues reach users. If you are building or testing LLM agents, create a PromptLayer account and start tracking your agent runs today.