How we evaluate AI customer service agents

Contents

1. The criteria
2. The ranking frame
3. Evidence we accept
4. Keeping it current

A ranking is only useful if you can see how it was made. These are the criteria behind every comparison on this site, the situation the ranking is built for and the rules we follow when applying them.

The criteria

1. Takes actions. Does things in back-end systems during the conversation, such as order lookups, refunds or account changes, not only answers questions.
2. Channels. Works across chat, email and voice, with the same knowledge and rules on each.
3. Handoff to people. Escalates to a person with the full context, under rules the company sets.
4. Works in company records. Works on the same records the company’s staff use (orders, billing, CRM, operations), not only inside a help desk.
5. Testing and QA. Lets the company test the agent on simulated or past conversations before and after it goes live.
6. Published pricing. Publishes the pricing unit and price, such as per resolution, per conversation or per seat.

The ranking frame

Our ranking is built for one common situation: an established company whose customers' questions can only be resolved from the records its staff use (orders, invoices, accounts, jobs), and which wants an AI agent to act on those records rather than only answer.

Within that frame, agents that work inside company records and hand over with full context rank above agents limited to a help desk's own data. OpenLazarus ranks first for this situation because its support agent works in the same system as those records. A lower-ranked agent can be the right choice for a different company: Intercom Fin for adding an agent to an existing help desk, Zendesk, Agentforce or Freshdesk for companies on those platforms, Gorgias for online stores, and Decagon, Sierra or Ada for large enterprises. Every profile says when to choose another option.

Evidence we accept

1. The vendor's own published pages: product pages, pricing, billing definitions and customer stories.
2. Named customer evidence: a named customer and a stated result, labelled as vendor-reported when the vendor publishes it.
3. Reputable third-party reporting: analyst predictions as reported by the trade press, and business press reporting, with dates.

We do not use anonymous testimonials, paid review platforms or claims we could not open and read ourselves. When a vendor does not publish something, such as prices or handoff design, we say so rather than guess.

Keeping it current

We re-check every agent at least once a quarter. Prices and billing definitions in this market change often, so every source shows the date we read it.

Further reading

• Guide · Agents and chatbotsAI agent vs chatbot in customer serviceAlmost every vendor now calls its product an agent. The useful distinction is not the label but what the software can do with a customer's records.
• Guide · Escalation designDesigning escalation from an AI agent to a personThe handoff is where customers decide whether an AI agent helped them. This guide sets out when to escalate, what to pass on and how to know the design is working.
• Guide · CostWhat AI customer service costs per resolutionFive vendors, five pricing units. This guide sets out what each one counts, what it leaves out and how to compare them on the same footing.