Through early experimentation, meticulous measurement, and the development of an architecture that changes with every model, Intercom produced a scalable AI platform that releases new features in days rather than quarters.
When Intercom(opens in a new window) debuted ChatGPT in 2022, they mobilized rather than merely observing the news. The customer support software firm started experimenting within hours of GPT 3.5’s release, and four months later, they introduced Fin, their AI Agent, which currently answers millions of client inquiries per month.
It wasn’t a coincidence that we gained early momentum. Intercom realized that AI will change the customer experience as LLMs advanced. Quick action was taken by the leadership, who committed $100 million to replatforming the company around AI, canceled non-AI projects, and established a cross-functional task force.
Reorganized product teams, a new AI-first helpdesk strategy, and a platform designed to help Fin handle large volumes and intricate client inquiries were among the company-wide changes brought about by that choice.
The three lessons from Intercom’s journey listed below can be applied immediately by any team, regardless of where they are in their development.
“AI-first cannot be added; it must be integrated.”
Paul Adams, Intercom’s Chief Product Officer
Lesson 1: Develop Model Fluency By Experimenting Frequently And Early
Intercom learns a great deal from their work and tests models frequently and early.
The group started experimenting with generative models early on, and their practical experience enabled them to identify potential and map model constraints. They were prepared for the release of GPT-4 in early 2023. They debuted Fin in four months, and they haven’t slowed down since.
Jordan Neill, SVP of Engineering, states, “We were able to leverage GPT‑3.5 to have fluid conversations with glimpses of magic, but it wasn’t yet reliable enough to trust with our customers.” “We shipped Fin because we knew it was ready when GPT-4 arrived because we had done the work.”
Intercom was able to create Fin Tasks, a system that automates intricate processes like technical support and refunds, thanks to the same fluency. Although the researchers had originally intended to use a stack based on a reasoning model, their assessments revealed that GPT 4.1 could manage the task independently—with more dependability and reduced latency.
A substantial portion of Intercom’s AI usage today, including crucial logic within Fin Tasks, is powered by GPT 4.1. Additionally, the team found that performance gaps were eliminated by adding chain-of-thought prompting to non-reasoning questions.
The lesson from Intercom is that the more familiar you are with your models, the more quickly you can adjust as the state of the art changes.
In comparison to GPT-4o, GPT-4.1 had the highest reliability in task completion and a 20% cost savings in Intercom’s evaluations. To minimize variance, completeness metrics were averaged over five separate runs (using Pass@k); a result is only considered “complete” if it passes each of the five runs.
Lesson 2: Use Robust Evaluations To Unlock Speed
You must measure what works and why if you want to progress quickly.
Intercom’s thorough evaluation approach is the foundation of their rapid adoption of new models, modalities, and architectures. Before being deployed, each new OpenAI model is evaluated for instruction following, tool call accuracy, and overall coherence using structured offline tests and live A/B trials, regardless of whether it is used for Fin Voice, powered by the Realtime API, or for Fin Tasks, powered by GPT‑4.1.
To assess how successfully models manage multi-step instructions like refunds, preserve Fin’s brand language, and consistently execute function calls, for instance, the team compares them to transcripts of real support contacts. Live A/B tests that evaluate customer satisfaction and resolution rates among models such as GPT-4 and GPT-4.1 are informed by these findings.
Intercom was able to move from GPT-4 to GPT-4.1 in a matter of days thanks to this method. They implemented GPT 4.1 across Fin Tasks after verifying enhancements in function execution and instruction processing, and they immediately observed increases in user satisfaction and performance.
“Within 48 hours of GPT 4.1’s release, we had evaluation results and a rollout strategy,” explains Pedro Tabacof, Intercom’s Principal Machine Learning Scientist. “We saw right away that GPT 4.1 had a good balance between latency and intelligence for our customers’ needs.”
The same assessment procedure assisted Intercom in validating new speech model snapshots for Fin speech and identifying enhancements in latency, function execution, and script adherence—all crucial for providing phone support that is human-quality.
In order to capture the additional dimension that speech adds to encounters, Intercom extended its evaluations. To guarantee excellent client experiences, they methodically evaluate Fin Voice for elements like personality, tone, handling interruptions, and background noise.
Lesson 3: Use Architectural Flexibility To Provide Long-Term Benefits
Since its inception, Intercom has been adaptable, creating an architecture that can change with the models it relies on.
With distinct choices for latency and complexity, Fin’s system is modular by design and supports a variety of modalities, including voice, email, and chat. Without reengineering the underlying system, the architecture enables Intercom to switch models and redirect inquiries to the most appropriate model for the task.
This adaptability is intentional and ever-changing. Currently in its third major iteration, Fin’s architecture is already being developed for the next one. As models advance, the team simplifies where it can and adds complexity where necessary to unleash new capabilities.
With Fin Tasks, this flexibility was essential. The team first believed that in order to implement Fin Tasks—which allow Fin to handle multi-step procedures like providing refunds, changing accounts, or addressing technical issues—they would require reasoning-based models.
However, in tests, GPT-4.1’s capacity to follow instructions performed better than anticipated, offering the same reliability at a reduced cost and latency.
Pratik Bothra, Principal Machine Learning Engineer at Intercom, adds, “To be honest, I don’t think people talk about GPT‑4.1 enough.” “The latency and pricing profile truly caught us off guard. It enables us to simplify and change our architecture.
A modular sub-agent architecture is depicted in the “Intercom AI Engine Diagram” flowchart diagram. It displays a question that has gone through six steps, each driven by a specific LLM: vector search, custom chunking, custom re-rankers, refine, create, and validate. To generate a final response, the flow places a strong emphasis on retrieval, reranking, and multi-stage validation.
Fin AI EngineTM
Using Workflow Automation And Unified Data To Create Connected Customer Experiences
The group is only beginning. With the help of sophisticated models and a modular, model-agnostic architecture, Intercom is going beyond customer service to power corporate activities, improving customer experiences and resolving issues more quickly:
- Support teams: Using Fin AI Agents to answer most incoming questions via chat, email, voice, and other channels
- Operations teams: Using Fin Tasks to automate intricate processes like account modifications, reimbursements, and subscription updates
- Product teams: Teams throughout the company may identify issues, create roadmaps, improve messaging, and get ready for QBRs by using AI tools like ChatGPT to access customer conversations, tickets, and user data via Intercom’s MCP Server.
By being performance-based, rigorous in review, and adaptable in design, Intercom created a scalable AI platform that redefined assistance and provided insights for any business utilizing AI.