Building a Real-Time Agentic Copilot for a Bank 

{{info-banner}}

Company profile

One of the largest universal banks in the region, serving retail, SME, and corporate clients across a broad branch and digital footprint. Its contact center sits at the intersection of customer experience and regulatory obligation - every conversation has to be fast, accurate, and defensible.

At the scale a bank operates, even small inefficiencies per call compound into high cost, risk, and CX consequences across millions of interactions per year.

The challenge

Before working with Flexy, contact center agents operated without a live intelligence layer during calls. Each conversation depended on the agent's memory, manual navigation across internal systems, and whatever knowledge they could recall in the moment.

This created several compounding problems:

• No live semantic understanding of the call. Agents had to interpret intent and track context entirely on their own, while also speaking with the customer.
• Fragmented access to knowledge. Policies, product rules, and procedural documents sat across multiple internal systems, forcing mid-call searching that added seconds (and sometimes minutes) to every interaction.
• Inconsistent resolution paths. Two agents handling the same scenario could land on different answers, depending on experience, shift, and team.
• High cognitive load per call. Agents were simultaneously listening, classifying, searching, deciding, and speaking - a workload that degrades under volume.

The result was longer handling times, avoidable escalations, and variable service quality that was difficult to track, improve, or enforce.

The challenge wasn't replacing agents or layering another post-call QA tool on top. It was building a real-time decision-support system that works inside the call itself.

Solution

Flexy designed and built a real-time voice agent system that embeds directly into the live conversation, turning every call into a structured, machine-interpretable workflow as it happens.

Instead of relying on post-call analytics or passive knowledge bases, the system operates as a live copilot layer: it transcribes, interprets, retrieves, and suggests - continuously - while the agent speaks with the customer.

What the system does during a live call

1. Transcribes the conversation in real time using fine-tuned NVIDIA streaming speech-to-text models.
2. Segments speakers through diarization so customer and agent utterances are tracked separately.
3. Extracts intent and state continuously via an LLM reasoning layer, updating as the conversation evolves.
4. Retrieves grounded context through a RAG layer over the banks' internal banking knowledge systems.
5. Suggests resolution actions to the agent inline, tied to the current intent and supported by source documents.

The design pattern is a supervisor-style orchestration over a real-time agent pipeline - stream, interpret, retrieve, suggest - built to run entirely on local infrastructure for regulatory compliance.

Flexy's approach

Flexy treated this as an agent workflow and infrastructure problem, not a generic LLM deployment. The objective was a system that agents would actually rely on mid-call, which meant sub-second responsiveness, tight grounding, and a UI that reduced cognitive load rather than adding to it.

Discovery (workflow mapping). Flexy mapped how calls actually unfold - the common intents, the points where agents reach for documentation, the scenarios that drive escalations, and the moments of cognitive overload. This framed what the copilot had to do, and what it had to stay out of the way of.

Architecture (real-time agent pipeline). Next came the system blueprint: a streaming ASR layer feeding a diarization and intent-tracking layer, a retrieval layer grounded in Banks internal knowledge, and an orchestration layer that coordinates suggestions without blocking the live conversation. Latency budgets were defined per stage.

Deployment (local inference, regulatory-grade). To meet banking data-residency and compliance requirements, the entire system was deployed on an 8× NVIDIA H100 cluster inside the bank's environment. No customer conversation leaves the bank's perimeter.

Experience design (the agent's screen). Flexy designed the interface so agents see a live transcription stream, a continuously updated intent label, and contextually retrieved resolution suggestions - organized so the agent can glance, not read, while staying in the call.

Technical details

• We used Nvidia NeMo framework for voice processing, LLM orchestration, and tool calling
• Neo4J for a knowledge graph
• PGvector for semantic search and RAG
• FAST API and Postgres were used for backend services
• React.js in the frontend

Deliverables

• Real-time voice agent system deployed on local 8× H100 infrastructure
• Streaming speech-to-text pipeline with diarization, fine-tuned on banking conversational data
• LLM-based intent and state-tracking layer that updates continuously through the call
• RAG retrieval layer over the bank's internal policy, product, and procedural knowledge
• Tool orchestration layer for surfacing resolution actions in-call
• Agent-facing UI designed for low-cognitive-load use during live conversations
• Monitoring and evaluation pipeline for ongoing quality and drift tracking

The results

The copilot turns each call from a memory-and-search exercise into a guided workflow. Agents spend less time navigating systems and more time resolving the customer's actual issue - and the resolutions themselves become more consistent across the floor.

• Reduced average handling time through faster in-call decision support
• Higher consistency of resolutions across agents, teams, and shifts
• Lower cognitive load per call, as retrieval and intent tracking happen automatically
• Reduced dependency on manual document search during live interactions
• Full data residency preserved through local inference - no customer audio leaves the bank

Differentiator

Most "AI for contact centers" tooling is built around post-call analytics - quality scoring, compliance review, and coaching after the fact. Flexy's system for this client has been fundamentally different: it is a real-time agentic decision layer embedded inside live customer interactions, producing structured intent, grounded retrieval, and resolution suggestions in the same moment the agent needs them.

It does not analyze the call after it ends. It participates in the call while it is happening.

Flexy is grateful to the client for their trust and collaboration on a project that sits at the intersection of customer experience, regulatory rigor, and applied AI.

‍

Contact us, and let’s start exploring your idea.