How should a fintech team evaluate an AI vendor?
Evaluate the vendor's failure behaviour, not their accuracy claim. Ask for the worst ten calls they can show you, what the system did when it was wrong, and who found out. A vendor who can answer that has built something; a vendor who answers with a benchmark number has built a demo.
The demo is a different product from the system
A vendor demo is the highest-performing instance of their system that will ever exist, chosen by the people who best know how to feed it. Every hour of your evaluation afterwards is spent discovering the distance between that instance and ordinary Tuesday traffic. So the useful questions are all about that distance rather than about the demo itself.
Five questions worth asking
"Show me your ten worst examples." A team that has shipped has failures they can replay. If the worst they can produce is a mildly awkward transcript, they have not run this in anger.
"What did the system do when it was wrong?" There is a right answer here and it is usually boring: it noticed a limit, it stopped, it handed to a human, it logged. The bad answer is that it completed the task confidently and someone found out later.
"Who found out?" Errors discovered by your own operations team before the customer are managed. Errors discovered by the customer are a relationship event, and in consumer finance they are also a compliance one.
"What does the integration actually touch?" Map the write path. Anything that writes to a ledger, moves money, changes a customer's status, or sends a message on your brand's name needs a limit you can point at in their documentation and in your own controls.
"What happens when you change it?" A model swap, a prompt edit, a price change. In fintech an AI change that looks small is a UAT event, because the acceptance criteria belong to the business process and not to the vendor's release notes. If the vendor has no answer, you are holding all the regression risk yourself, quietly, every time they ship.
Run one real scenario end to end
Before signing anything, take one live customer scenario and run it through the vendor's system with your own data, your own agents, and your own definition of a good outcome. One is enough, and it should be the boring one, not the impressive one — the daily task that the team currently does hundreds of times a week. You are testing the integration and the handoff, which is where deals fail, not the model, which is where the demo already won.
This is the same discipline I apply to launch readiness on my own projects: real scenarios, system limits, and operational readiness before users touch it. Buying an AI system and building one fail in the same place, and it is not the model.
Filbert Henrico is a Technical Customer Success Lead at FINETIKS in Tangerang since June 2025, with seven-plus years of customer success and technical sales across Southeast Asian fintech, including Xendit, Peakflo, and Brick. His focus includes API integrations, agentic AI, workflow automation, conversational AI, UAT, and launch readiness. Bachelor of Marketing, Victoria University, Bandar Sunway, Malaysia, 2016–2019.
Stuck on something similar?
These pages describe how I approach the work, not a fixed methodology. If you are scoping something similar and want a second opinion on the trade-offs, send me the problem in the messy state it is actually in.
Keep reading
All questionsWhich AI use case should a team build first?
Build the one where a human already does the task every day, the correct answer is easy to recognise, and a wrong answer is cheap to undo. Those three conditions are what let you finish the project: the workflow gives you data, the recognisable answer gives you a test, and the cheap undo gives you permission to ship.
How do you automate customer support without breaking escalation?
Map every escalation rule a human currently applies, and automate only the work that sits above the rules. Escalation is not a feature you add to an automated flow — it is the boundary of the flow, so the boundary has to exist and be written down before anything is automated. Teams that automate first and define escalation later end up with a system that handles everything confidently, including the things it must not touch.
What does production readiness mean for a voice AI agent?
A voice agent is production-ready when it can end a call safely without ever needing to be clever. That means four things are tested against real audio, not transcripts: it recognises the caller's actual speech, it handles being interrupted, it has somewhere to hand off to a human, and every action it takes against a real system has a limit it cannot exceed.