Would the new AI agent handle the same refund better? Test it before rollout
NiCE Cognigy's multivariate testing compares contact-center agents before deployment. Use refund, cancellation and handoff scenarios to find what a smooth demo can miss.
A customer calls because a return arrived at the warehouse eight days ago and the refund still has not appeared. One AI agent promises the money will arrive tomorrow. A second checks the order, explains that a refund was started but not settled, and transfers the call when the customer disputes the amount. Both sound competent. Only the order system and the handoff record can tell you what actually happened.
On November 12, 2026, contact-center software provider NiCE Cognigy introduced a new version of its agent platform. Its Multivariate Testing feature runs simulations side by side using different agent versions, language models, agents or scenarios. The announcement also describes agent traces for inspecting skills, tool calls and model usage. The new environment became available in trial accounts on November 12, with a gradual rollout to existing customers starting in November. NiCE Cognigy product update
The useful first comparison is two versions of the same agent on the same customer problem. A refund, a cancellation and a request for a person show which outcomes to verify before you put either version in front of callers. Start with the behavior the customer should experience and check the system state that supports it.
A conversation needs a testable outcome
The refund example has several possible correct answers. If a payment is still pending, the agent can report that state and explain where to check. If the returned item was received but no refund started, the agent may need to follow an existing policy or hand over to a team with the right permission. If the caller cannot establish access to the account, the agent should avoid disclosing an order status. A single scenario called “refund inquiry” can hide all three decisions.
Give each test a clear starting record and a permitted ending state. In one case, a fixture says the refund is pending and already has a reference number. A passing agent reads the status and creates no second refund. In a separate case, the system returns an error when the agent looks up the order. A passing agent explains that it could not confirm the status and routes the request through the available support path. The expected outcome comes from your business policy and system state.
The caller's language still matters. “I just want this over with” could be a request for a refund status, an invitation to cancel something, or frustration with waiting. Include variations of the same underlying case so the test does not reward the exact wording in a single transcript. Use both a direct instruction to transfer and a softer request such as “Could someone look at this with me?”
Hold the customer case steady and compare the conversation and system outcome together. Original illustration by Telli.sh.
For a live contact center, it is also useful to distinguish a customer asking a question from a customer authorizing an action. A person might ask what a cancellation would cost and then choose to keep the service. A good test checks that the agent can quote a supported amount and leave the account active. This becomes especially important when a tool call makes a visible change that a human will have to reverse.
| Case | What should happen | Evidence for the reviewer |
|---|---|---|
| Refund already pending | Explain the current state without creating a duplicate | Order status, tool-call record and transcript |
| Customer asks about a cancellation fee | Present the fee or transfer when it cannot be verified | Source policy and absence of an account change |
| Customer explicitly asks for a person | Route to an appropriate queue and send permitted context | Routing event and the receiving agent's handoff packet |
| Order lookup is unavailable | Give an accurate status of the lookup and offer a next step | Tool error, user-facing wording and any retry or transfer record |
Side-by-side testing is easiest to interpret when one thing changes
Cognigy's update describes comparisons across Snapshots, models, agents and scenarios. Begin by varying only one. If you change both the model and its instructions, a better answer will tell you little about which change improved it. If you change a transfer flow and a model together, a routing regression can be hard to attribute. One control makes the result easier to act on. NiCE Cognigy product update
A practical first run holds the conversation policy and tools fixed while comparing the existing model with a candidate model. Once one version looks promising, compare two prompt or flow configurations on that model. Use the same order records, time assumptions and fixtures each time. Make the state fresh for every run so an earlier test does not leave behind a closed case that changes what a later test sees.
Test interactions can be variable even with fixed inputs. A model may make a different choice on a second run, especially if the fixture allows several valid routes. Repeat cases involving payments, consent or handoffs enough to reveal obvious instability. Record counts with the denominators. One failed handoff in three tests and one in a hundred tests deserve different interpretations, and neither establishes a live failure rate.
Trace information makes a regression more actionable. Suppose the agent says “I checked your refund” but the trace shows no order lookup. That is a grounding problem. Suppose the correct lookup ran, returned “pending” and was summarized as “paid.” That is a reading or interpretation problem. The two errors produce similar messages to a customer, but they lead to different repairs. NiCE Cognigy lists skill execution, tool calls, model calls and token usage among the trace details. NiCE Cognigy product update
Keep business rules in the part you can verify
Cognigy says existing customers can retain Flows and Flow Agents, combine them with the new AI Agents or adopt the newer agent-led architecture. A flow can remain the owner of a deterministic step, such as checking consent, and an agent can handle the conversational steps around it. The release also describes passing control back into a flow. The design gives a team room to update a single behavior while keeping established routes in use. NiCE Cognigy product update
Picture a caller who asks for a refund, then asks for a person while the agent is still talking. A conversational model can interpret the shift. A fixed handoff flow can choose the queue and pass the contact reason and the verified account identifier when allowed. A test can inspect the receiving side and establish whether the transfer really happened. An assistant saying “I am transferring you” is an intermediate signal and the queue event is evidence of the transfer.
Now give the same call a different ending. The appropriate queue is closed and the agent receives a tool response that only says the transfer was unavailable. Review the trace in order. It should show whether the agent chose the expected queue, whether the destination rejected the request and whether another permitted support route was offered. An agent that accurately explains the situation may pass the communication check while the routing setup still needs work. If the tool response reports success but no ticket reaches the queue, the support team has a different problem to investigate.
This is why it helps to use a status with a meaning that everyone on the team agrees on. “Transfer requested” says the workflow tried. “Transfer accepted” says the destination received it. “Handoff complete” should mean a receiving agent or approved queue obtained the promised context. Record the last state you can verify and make the test expectation explicit when a queue is closed.
Pair the example with a constrained cancellation task. Before changing the account, the workflow can require a clear confirmation after the customer sees the applicable terms. A simulated call should stop before the operation when the caller merely asks for a quote. It should also stop when a prerequisite fails. This is a useful place to assess the conversation and the action independently, since a polite explanation can still accompany an incorrect state change.
Compare flexible conversation with the business checks that govern consequential actions. Original illustration by Telli.sh.
Simulated and live contacts answer different questions
Simulations are useful for replaying the same case across configurations. They make a rare but important failure easy to test again. Production calls introduce real wait times, background noise, interruptions and changes in the tools or staffing that a fixture may not reproduce. An unusually clean test call tells you what happens in a controlled setting. Design a live observation plan to see how the same behavior holds up in ordinary use.
For voice, listen for how long the caller waits before the next meaningful response and whether the system stops speaking when interrupted. A technically correct account answer can feel unhelpful if the agent talks over a correction. Test short and long pauses, an accented variation of the same request and a caller who switches from a status question to a transfer. Cognigy's announcement lists real-time voice-to-voice support with turn-taking and tool use among its features. Your phone routing and acoustic conditions will shape what callers experience. NiCE Cognigy product update
For chat, review what happens across a session break. A customer may go looking for an order number and return later. A follow-up should use the right context when available and ask again when the session no longer establishes the information. Compare these outcomes with the same policy applied to both agent versions. The exercise matters for channels where a long gap is ordinary behavior.
Choose at least one production measure for each decision the simulation is meant to improve. A lower transfer count can mean the agent solved more problems, or it can mean callers struggled to reach a person. Compare repeat contacts, outcomes in the system and handoff success alongside the transfer rate. Check how cases are sampled so a change in call mix does not get credited to a configuration update.
Start with a test you could explain to the receiving agent
- In a trial or eligible customer environment, choose one customer journey and an existing configuration as the baseline. Record exactly what changes in the candidate. Use synthetic or authorized data and test integrations for consequential operations.
- Prepare repeatable starting records for four cases. Include an already pending refund, an unavailable lookup, a question about cancellation and a direct request for a person. Write the expected final states before opening either transcript.
- Run the same fixtures against both configurations in Multivariate Testing where available. Review the utterances, tool events and final system states together. Reset fixtures between runs and repeat cases that are sensitive to wording or timing.
- Treat errors that disclose the wrong information, change an account without authorization or silently lose a requested handoff as distinct from a wording issue. Trace the first unexpected step and adjust its owner before rerunning that specific scenario.
- Establish a limited observation period for eligible live traffic. Compare the same outcomes and inspect examples with operators who receive the transfers. Keep an easy way to return to the baseline if an important behavior regresses.
For an enterprise team already using Cognigy, the product update says production environments are receiving the features gradually and customers will be notified. Start with the trial if you want to see the design and verify access in the intended environment before planning a live comparison. NiCE Cognigy availability and trial details
A better contact-center agent leaves an intelligible record of what it said and what it did. If you can show the refund status, the authorization step and the receiving queue for the same test, you have a comparison a support team can use to decide which version belongs in front of customers.
Sources
- NiCE Cognigy's next-generation platform announcement, November 12, 2026, with screenshots and rollout details
- NiCE Cognigy company and product background
To keep transcripts and decisions from support reviews searchable, create a Telli.sh account.