Enterprise IT teams are rethinking how they prevent system failures, resolve incidents, and manage growing workloads. Discover how AI-powered operations can reduce alert fatigue, accelerate root cause analysis, and free engineers to focus on business-critical innovation.
Introduction: The Growing Cost of IT Failures
When a business-critical system fails, every minute of downtime can mean lost revenue, frustrated customers, and damaged trust. As enterprise technology becomes more complex, IT systems generate alerts and operational data faster than human teams can realistically process.
The consequences can be significant. Following a global IT outage in July 2024 caused by a faulty software update, a major U.S. airline reported approximately $380 million in lost revenue and $170 million in additional costs associated with roughly 7,000 flight cancellations over five days.
Alert overload presents another challenge. A study of a large commercial bank, presented at the 2020 International Conference on Software Engineering, documented alert storms involving hundreds to thousands of notifications per minute. These events occurred approximately weekly and required an average of six engineers working for 55 minutes to manage. Among 44 surveyed engineers, 77% reported receiving too many alerts to identify problems manually.
These examples highlight a fundamental problem: enterprise IT teams are not simply dealing with more incidents. They are dealing with more data, increasingly interconnected systems, and less time to determine what matters.
In a series of episodes on the AI in Business Podcast, Emerj explored how enterprise leaders can modernize IT operations through intelligent automation, improved incident management, and proactive failure prevention.
The discussions featured Assaf Resnick, Founder and CEO of BigPanda; Luke Rotta, Director of Site Reliability Engineering at Charles Schwab; Dave Glick, Senior Vice President of Enterprise Business Services at Walmart; and Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase.
Across their conversations, four strategies emerged: building enterprise-owned operational knowledge, correlating alerts across applications, automating repetitive tasks, and eliminating recurring problems instead of continually adding support staff.
1. Build an Enterprise-Owned Knowledge Graph to Resolve Incidents Faster
One of the biggest obstacles to efficient IT operations is the loss of institutional knowledge.
Modern enterprises rely on hybrid cloud environments, microservices, and continuous software releases. These technologies create increasingly complex systems that change frequently and generate enormous volumes of operational data.
Assaf Resnick argues that organizations often treat IT operations as a staffing problem when the underlying challenge is access to relevant information and knowledge.
Traditional outsourcing and shift-based support arrangements can make this problem worse. Engineers may leave after gaining valuable experience, while handoffs between teams can cause important context to disappear. Knowledge about legacy applications may also remain undocumented, residing primarily with experienced employees.
Resnick proposes addressing this challenge through an enterprise-owned IT knowledge graph that gives AI agents the context they need to investigate and resolve incidents.
Such a knowledge graph should bring together three essential types of information:
- Live operational awareness: Real-time information about system performance, alerts, dependencies, and changes across the IT environment.
- Historical incident data: Records of previous failures, troubleshooting steps, incident discussions, and successful resolutions.
- Institutional knowledge: Undocumented expertise from experienced operators and insights learned from observing how skilled engineers solve problems.
Consider a hypothetical scenario involving a bank’s ATM network. Thousands of alerts could overwhelm an inexperienced support team, delaying the identification of the actual cause. Investigations might continue for hours before someone discovers that a security update introduced the problem weeks earlier.
With access to a knowledge graph, an AI agent could recognize a similar historical incident, identify the potentially related change, and recommend an appropriate corrective action much faster.
However, AI should not automatically handle every incident. When confidence is low, or the situation requires specialized judgment, the agent should escalate the issue to the appropriate human operator or technical specialist.
The key is to introduce AI gradually rather than attempting to replace every existing process at once. Organizations can begin with recurring, lower-risk workflows, demonstrate measurable improvements, and expand automation as confidence grows.
Equally important, the operational knowledge must remain under the enterprise’s control rather than becoming inaccessible within a vendor’s proprietary platform.
2. Use AI to Correlate Alerts and Identify Root Causes
Generating more monitoring data does not necessarily make IT operations more effective. The real challenge is understanding how individual alerts relate to one another and determining where a problem originated.
Dhrubojyoti Das Deb explains that an error appearing in one application may actually originate somewhere else in a chain of connected systems.
For example, an issue in the first application might pass incorrect information through two additional applications before a fourth application generates an error. Engineers investigating only the application that reported the failure could spend valuable time treating symptoms rather than addressing the underlying cause.
Several factors make this increasingly difficult:
Alert overload. Large IT environments generate substantial amounts of low-value information, including debugging messages and informational notifications. Das Deb estimates that 99% of incoming data may be noise in some operational contexts.
Complex application dependencies. Enterprises operate applications built with different programming languages, platforms, cloud services, and AI technologies. Tracing a problem across these interconnected components requires a broader view of the environment.
Continuous system changes. Frequent software releases and configuration updates can introduce new problems before monitoring and traceability systems have fully captured the changes.
AI can help by grouping related alerts into a smaller number of meaningful incidents and correlating those incidents with logs, traces, metrics, deployment histories, and code changes.
Instead of asking engineers to investigate thousands of individual notifications, an intelligent system can identify patterns and highlight the incidents most likely to require attention.
Das Deb describes the value of this approach as reducing thousands of messages to a small number of distinct incidents. AI performs the initial analysis, while human experts validate the findings and make important decisions.
This creates a more efficient division of responsibility: machines process large volumes of information, and people apply judgment, experience, and accountability.
Luke Rotta also emphasizes the importance of providing responders with the right context. During an incident, teams need to understand five things:
- What changed?
- What is broken?
- Why is it not working?
- Who should be contacted?
- What happened the last time a similar problem occurred?
Answering these questions quickly can significantly improve incident response.
Why Data Quality Still Matters
AI-driven incident management depends on reliable, structured operational data.
Logs without timestamps, application names, module identifiers, or meaningful error context can make it difficult for an AI system to determine where a problem began. Legacy production environments may therefore require improvements to logging, monitoring, and observability before advanced automation can deliver its full potential.
The objective is not simply to introduce another AI tool. It is to build a connected operational environment in which relevant information can be interpreted and acted upon.
3. Automate High-Volume Tasks Before High-Risk Tasks
Not every IT automation opportunity needs to involve a major system failure.
Luke Rotta recommends prioritizing repetitive, high-volume operational tasks before attempting to automate the most complex or highest-risk scenarios.
Small problems can consume enormous amounts of time when they occur repeatedly. Examples include manually resolving server disk-space issues, processing routine change requests, managing releases, and investigating familiar alerts.
Although each task may appear insignificant, their cumulative impact can reduce team capacity, slow innovation, and contribute to employee frustration.
Rotta describes this as a “death by a thousand cuts,” in which numerous minor tasks gradually drain an organization’s time and attention.
AI-powered automation can identify known patterns, execute approved corrective actions, document what it has done, and escalate situations it cannot safely resolve.
This approach also creates an opportunity to reinvest the time saved into further automation.
Dave Glick describes a similar pattern at Walmart. Many AI demonstrations during his organization’s weekly AI office hours focus on reducing operational toil rather than developing new production features.
For example, tasks such as preparing weekly ticket summaries can be completed in minutes with AI, revealing recurring issues that would otherwise require significant manual effort to identify.
The practical challenge is breaking the cycle in which teams are too busy handling repetitive work to improve the systems responsible for generating it.
How to Choose the Right Automation Opportunities
Organizations can begin by documenting operational friction across their teams and prioritizing tasks based on volume, risk, complexity, and implementation effort.
A practical starting point includes:
- Identify repetitive work. Review frequent alerts, recurring service requests, and routine manual interventions.
- Start with manageable tasks. Choose workflows that have predictable outcomes and clearly defined procedures.
- Test before deploying. Run AI alongside existing human-led processes and compare the results before granting it greater responsibility.
- Measure the impact. Track time saved, incident resolution speed, escalation rates, and the number of recurring tasks eliminated.
- Expand gradually. Use successful early implementations to establish confidence and build a roadmap for additional automation.
Rotta also suggests examining whether existing alerts genuinely require action. In one example, he described temporarily disabling alerting to determine whether the notifications were providing meaningful value. Such an experiment may not be appropriate for every organization, especially in heavily regulated environments, but it illustrates the importance of questioning unnecessary operational noise.
A safer starting point for many enterprises is to retain visibility into raw alerts while prioritizing responses based on meaningful service-level objectives and indicators.
The broader lesson is to automate deliberately, beginning with tasks that offer clear benefits and manageable risks.
4. Eliminate Recurring Tickets Instead of Adding More Support Staff
When support requests increase, adding more personnel may seem like the simplest solution. However, hiring additional Level 1 and Level 2 support staff does not necessarily address the underlying problems creating those requests.
Dave Glick advocates a different approach: eliminate the problem whenever possible, automate what remains, and optimize the process afterward.
As he puts it, “The best ticket is no ticket.”
Every recurring support ticket consumes time that engineers could otherwise spend building features, improving customer experiences, reducing costs, or supporting business growth.
Glick refers to many of these issues as paper cuts: small, persistent problems that create unnecessary friction for customers and employees.
For example, sellers on a platform repeatedly encountered errors because of an image-upload limit that was difficult to explain. Increasing the limit required only a small code change, but addressing it could reduce repeated errors and improve the seller experience.
The solution was not to create a more efficient process for handling complaints. It was to remove the source of the complaints.
A Practical Approach to Ticket Elimination
Glick’s approach involves three core practices.
Maintain a list of recurring problems. Document minor bugs, frustrating limitations, and unresolved issues that frequently fall behind larger development priorities. A running list helps ensure these problems receive attention.
Fix small issues consistently. Rather than postponing every minor improvement until the next quarterly planning cycle, address a manageable number of issues each week. Small fixes can produce meaningful cumulative improvements over time.
Use AI to accelerate potential fixes. Coding agents can review documented issues, draft proposed code changes, and help generate test suites for engineers to evaluate. Human review remains essential before changes are deployed.
Glick also described using a coding agent to develop a test suite for a careers website launch in approximately a week, a task that would have taken months to complete manually. The tool helped the team address untested edge cases while working through a backlog of bugs.
These examples demonstrate how AI can help organizations address operational problems that might otherwise remain unresolved indefinitely.
The same principle applies to incident prevention. If a failure repeatedly occurs after a particular type of change, organizations should investigate how to prevent that change from causing the problem again.
Potential approaches include reviewing changes before deployment, identifying risky configurations, and using historical incident data to detect patterns associated with previous failures.
By eliminating recurring issues at their source, organizations can reduce ticket volume and redirect engineering resources toward higher-value work.
5. Establish Governance and Human Accountability
Although AI can improve IT operations, successful implementation requires more than technical capability.
Enterprise leaders must ensure that automated systems operate within established security, privacy, compliance, and risk-management requirements.
The podcast discussions highlighted three important practices for introducing automation responsibly.
Validate AI performance before deployment. Run AI alongside existing human-led processes and compare its recommendations with actual outcomes. Simulated environments can also help teams test behavior before introducing automation into production systems.
Keep accountability with people. AI may analyze data and recommend actions, but organizations remain responsible for the decisions they make and the outcomes customers experience. Teams must understand the automation they deploy and the risks associated with it.
Apply existing governance controls. Security reviews, privacy policies, compliance requirements, and release-management procedures should extend to AI agents. Organizations should also maintain visibility into which agents exist, what they are authorized to do, and how they are deployed.
A phased approach allows enterprises to develop confidence in AI without handing over control prematurely.
Initially, AI may recommend actions for human approval. As reliability improves and performance is validated, organizations can consider allowing it to execute specific, predefined actions under appropriate safeguards.
Full automation and self-healing operations should be treated as longer-term goals rather than assumptions about what AI can safely accomplish from the beginning.
6. What Enterprise Leaders Should Do Next
The experiences shared by Resnick, Rotta, Glick, and Das Deb point toward a common strategy for modernizing enterprise IT operations.
Instead of responding to growing complexity by continually adding people and tools, organizations can use AI to make their existing operational knowledge, systems, and processes more effective.
Leaders can begin by focusing on four priorities:
- Preserve institutional knowledge. Consolidate operational history, system information, and undocumented expertise into a knowledge foundation that the enterprise owns.
- Improve incident visibility. Connect alerts, logs, deployment information, and application dependencies so teams can identify root causes more quickly.
- Automate repetitive work. Target high-volume, predictable tasks that consume time without requiring extensive human judgment.
- Eliminate recurring problems. Investigate repeated tickets and incidents, fix their underlying causes, and prevent avoidable failures from returning.
Organizations should also define measurable outcomes before expanding their AI initiatives. Useful indicators include mean time to resolution, alert volume, recurring incident frequency, manual workload, service availability, and the amount of engineering capacity recovered.
These measurements help distinguish genuine operational improvements from automation that merely adds another layer of complexity.
Conclusion: From Reactive IT Support to Intelligent Operations
Enterprise IT operations are entering a period in which the ability to interpret information and act on it intelligently may matter more than simply increasing staffing levels.
AI offers organizations new ways to connect operational knowledge, reduce alert overload, accelerate root cause analysis, automate repetitive tasks, and prevent recurring failures.
However, the greatest benefits do not come from automating everything at once. They come from identifying the right problems, improving the quality of operational data, validating AI performance, and expanding automation under appropriate human oversight.
The message from enterprise leaders is clear: organizations should stop treating every incident as an isolated event and every growing workload as a reason to add more staff.
By combining AI-driven intelligence with human judgment and strong governance, enterprises can build IT operations that are more proactive, efficient, resilient, and aligned with business goals.
The ultimate objective is not simply to resolve incidents faster. It is to prevent avoidable incidents, eliminate unnecessary work, and give technology teams more time to create lasting business value.