Transforming Enterprise IT Operations: How AI and Agentic Automation Are Replacing Manual Triaging to Protect Revenue and Customer Trust

Modern enterprise systems operate at a scale and velocity that routinely exceed human cognitive limits. Driven by the rapid proliferation of hybrid cloud infrastructures, distributed microservices, and continuous integration and continuous delivery (CI/CD) pipelines, organizations generate operational data at rates never before seen. This hyper-acceleration of technology has fundamentally broken legacy incident management frameworks. Every single business-critical service failure now acts as a high-stakes timer ticking down toward irreversible losses in revenue, operational efficiency, and hard-earned customer trust.

The fragility of modern enterprise IT is no longer a theoretical risk; it carries catastrophic financial and operational consequences. A stark reminder of this vulnerability occurred in July 2024, when a single faulty software update triggered a massive global IT outage. In the wake of the incident, a major U.S. airline disclosed in regulatory filings submitted to the Securities and Exchange Commission (SEC) that it suffered approximately $380 million in direct lost revenue. Compounding this financial blow, the carrier incurred an additional $170 million in supplementary operational costs stemming from the cancellation of roughly 7,000 flights across a punishing five-day period.

Beneath such high-profile global catastrophes lies a daily, grinding operational crisis: the perpetual "alert storm." Empirical research underscores the immense strain this places on enterprise engineering teams. A comprehensive study of a major commercial bank—presented at the 2020 International Conference on Software Engineering—revealed that the institution routinely experienced alert storms generating hundreds to thousands of distinct alerts per minute on a weekly basis. Each of these storms required an average of six engineers spending roughly 55 minutes apiece just to stabilize the environment. Among 44 enterprise engineers surveyed during the research, an overwhelming 77% stated categorically that their systems generated far too many alerts for human personnel to accurately identify and resolve problems manually. Most alarmingly, researchers observed a dangerous psychological defense mechanism among operational staff: engineers frequently ignored the overwhelming flood of alerts entirely, waiting passively until mounting customer complaints finally exposed the underlying system failure.

Recognizing this critical enterprise bottleneck, Emerj recently hosted an extensive, multi-part series on the AI in Business Podcast. The program examined how global enterprises can radically modernize their IT operations by transitioning away from reactive firefighting. Discussions focused heavily on preventing recurring system failures, synthesizing overwhelming alert noise into actionable intelligence, and automating repetitive Tier 1 (L1) workloads. The podcast featured a distinguished panel of industry leaders: Assaf Resnick, Founder and CEO of BigPanda; Luke Rotta, Director of Site Reliability Engineering at Charles Schwab; Dave Glick, Senior Vice President of Growth Technology at Walmart; and Dhrubojyoti Das Deb, Senior Vice President of Technology at JPMorganChase.

The insights synthesized from these industry veterans point toward an undeniable macroeconomic shift: the economics of IT operations are fundamentally pivoting away from scaling human headcount toward scaling machine intelligence.

The Evolution of IT Operations: From Staffing Deficits to Data Overload

To understand why traditional incident management is collapsing, enterprise leaders must first reframe the nature of the challenge. According to BigPanda Founder and CEO Assaf Resnick, organizations have chronically misdiagnosed IT operations as a human staffing problem when it has definitively become a data problem.

For the past thirty years, a primary cost-cutting strategy for global corporations has been outsourcing IT support functions to third-party global systems integrators (GSIs). However, this playbook has proven increasingly ineffective against modern software complexity. In many outsourced environments, the average tenure of an L1 engineer hovers around seven months. Personnel frequently resign or rotate out shortly after finally mastering the nuances of a complex enterprise ecosystem. Furthermore, maintaining 2D or 3D round-the-clock global coverage necessitates multiple daily shift handoffs—critical operational seams where vital technical context is routinely lost. As Dhrubojyoti Das Deb of JPMorganChase notes, this structural vulnerability means that institutional knowledge of legacy applications frequently departs the organization entirely alongside the subject matter experts who held it.

Resnick advocates for a radical paradigm shift: onboarding autonomous software agents into the enterprise ecosystem with the same rigorous methodology applied to human employees. This requires the establishment of an enterprise-owned IT knowledge graph capable of synthesizing three critical dimensions of operational context: historical incident data, real-time telemetry, and explicit topology maps of the software architecture.

To visualize the contrast, consider a hypothetical network failure crippling a major commercial bank’s automated teller machine (ATM) network. Under legacy operational models, thousands of disparate alerts cascade down upon an L1 team that may be completely unfamiliar with the underlying microservices. A triage target of 15 minutes routinely balloons into a two-hour ordeal. Consequently, a high-stress bridge call is convened, dragging on for six grueling hours before an exhausted engineer finally traces the root cause back to an unflagged security patch deployed three weeks prior.

By contrast, when an integrated enterprise knowledge graph and agentic layer are deployed, an autonomous agent can detect the anomaly within 30 seconds. The agent instantly cross-references the telemetry against historical incidents—identifying a nearly identical failure from seven months prior—and automatically recommends a targeted rollback of the problematic patch.

While Luke Rotta of Charles Schwab notes that while incidents frequently share recurring patterns, his 25 years of financial technology leadership have taught him that every operational failure possesses unique variances. Recognizing this, Resnick emphasizes that when an agent’s confidence score falls below a predefined operational threshold, it intelligently hands the problem off to a human specialist. Crucially, the system routes the issue directly to the specific domain owner rather than pulling dozens of uninterested personnel onto a chaotic emergency bridge call. Rotta emphasizes that assembling the right human expertise rapidly is precisely where traditional incident resolution suffers the most friction.

Financially, this shift completely transforms enterprise cost structures. Resnick points to operational models transitioning from roughly 300 outsourced operators down to an intelligent agentic layer overseen by two or three high-level Site Reliability Engineers (SREs). He highlights a Fortune 500 healthcare client that successfully mitigated internal incidents and self-inflicted outages to a degree valued at approximately half a billion dollars.

Nevertheless, Resnick cautions against disruptive "rip-and-replace" strategies. He advises CIOs to approach agentic adoption the same way enterprises migrated core workloads to the public cloud: one critical workflow at a time. Furthermore, Resnick issues a stark warning regarding data sovereignty, insisting that organizations must retain strict ownership of their operational knowledge graphs rather than surrendering them to vendor-controlled walled gardens.

Cross-Application Correlation and the Quest for Root Cause Analysis

The primary bottleneck in modern incident response is no longer a scarcity of data, but rather the immense difficulty of isolating where a cascading failure originated. As Dhrubojyoti Das Deb explains, while enterprise logging, tracing, and telemetry tools have expanded exponentially alongside business growth, individual application data remains locked within impenetrable silos.

In a modern enterprise transaction chain spanning five interconnected applications, an error frequently manifests visibly in the fourth system. Meanwhile, the actual point of failure occurred silently in the first system—which threw no alerts whatsoever, but steadily passed corrupt data downstream through the second and third applications. Consequently, engineering teams waste precious hours aggressively investigating the symptom-bearing application while the root cause remains unaddressed.

Luke Rotta frames the true cost of these diagnostic delays not merely in terms of raw downtime, but as catastrophic opportunity cost. When a revenue-generating service goes dark, the business permanently loses transactional volume. The financial damage scales directly with the traffic density of the exact hour the outage occurs. Dave Glick, Senior Vice President of Enterprise Business Services at Walmart, describes this tedious diagnostic hunting as treating enterprise engineers like "digital detectives"—a painful waste of human ingenuity that modern automation must eliminate.

Alongside the erosion of tribal knowledge, Das Deb identifies three persistent friction points that obscure signal from noise: tool proliferation, format inconsistency, and asynchronous time-stamping across disparate cloud environments. Rotta argues that adding unintegrated monitoring tools only compounds the chaos. Instead, Das Deb’s engineering teams enforce strict tool consolidation, restricting analysts to a maximum of three core platforms to maximize data correlation capabilities.

According to Rotta, an effective incident context layer must instantaneously answer five fundamental questions during a crisis: What changed? What is broken? Why is it failing? Who owns the domain? And what happened the last time a similar failure occurred?

To achieve this, Das Deb outlines a two-stage artificial intelligence architecture:

1. Automated Noise Reduction: AI ingests thousands of raw log messages and groups them into a handful of distinct, synthesized incident clusters.
2. Contextual Correlation: Machine learning models cross-reference these clusters against recent code deployments, infrastructure changes, and historical runbooks.

Resnick notes that this architecture relies on a collaborative fleet of specialized agents—dedicated database agents, network agents, and cloud infrastructure agents—investigating anomalies concurrently rather than relying on massive, generalized language models that struggle with hyper-specific enterprise telemetry.

The human workflow changes dramatically under this model. Instead of wading through 5,000 fragmented alerts to hunt for a needle in a digital haystack, human analysts review pre-packaged incident hypotheses curated by AI, exercising high-level judgment to authorize automated or manual remediations. Das Deb stresses that this operates as a continuous feedback loop: humans validate outputs, AI refines its internal models, patches are deployed, and fresh telemetry feeds back into the system. Full self-healing automation, he cautions, must be approached incrementally as trust in the underlying models matures. Furthermore, legacy production environments must be systematically retrofitted to ensure consistent, standardized data logging—such as standardized timestamps and rigorous module labeling—without which AI models cannot accurately trace algorithmic fault lines.

Volume-First Automation Priorities to Recover Team Capacity

For engineering leaders seeking to deploy automation effectively, Charles Schwab’s Luke Rotta advocates for a strict "volume-first, risk-second" prioritization framework. High-risk catastrophic failures are statistically rare and typically feature existing redundancy or safety nets. The true operational drain on enterprise culture is "death by a thousand cuts"—the endless tide of low-risk, highly repetitive manual tasks that slowly exhaust engineering morale and team capacity.

Rotta points out that major financial institutions still routinely rely on human operators to manually resolve archaic issues, such as clearing server disks that run out of space—a maintenance problem that has persisted for thirty years. Modern, AI-driven automation can effortlessly intercept these known patterns, execute remediation scripts, log the action, and escalate exclusively when an anomaly defies standard automated parameters.

Assaf Resnick highlights how agentic operations eliminate the exhausting operational runaround. In traditional environments, before an incident is even verified, human operators waste valuable minutes frantically phoning network, cloud, and development teams to ask if anyone else is experiencing anomalies. Agentic workflows eradicate this administrative friction entirely.

At Walmart, Dave Glick experiences a similar cultural transformation through internal AI initiatives. During weekly AI office hours, approximately 75% of engineering demonstrations focus on eliminating administrative toil—such as streamlining cumbersome change requests and release management procedures—rather than building customer-facing features. A weekly ticket summary that previously demanded hours from a dedicated administrative team is now generated by AI in mere minutes, simultaneously surfacing critical systemic patterns.

However, organizations must beware of what Rotta terms the "hamster wheel": operational teams are frequently buried under so much manual toil that they lack the dedicated bandwidth required to build the very automations that would set them free. To break this cycle, leaders must occasionally force the issue. Rotta shares an anecdotal experiment from his past where his organization deliberately shut off all non-critical alerting for a full 24 hours. The astonishing result? Absolutely nothing broke, business operations proceeded entirely unaffected, and the handful of issues that did surface cleanly illuminated which alerts were genuinely critical. While heavily regulated institutions may lack the institutional risk appetite for such aggressive experimentation, Rotta recommends a pragmatic alternative: maintain raw alert reporting for compliance, but design human operational responses strictly around service level objectives (SLOs) and service level indicators (SLIs).

For deciding where to initiate automation projects, Dhrubojyoti Das Deb recommends compiling a comprehensive inventory of all operational friction points, ranking them by risk and implementation ease, and ruthlessly automating the easiest operational headaches first. Successfully automating just two out of ten friction points builds the organizational confidence and technical momentum required to tackle the remaining eight.

Across the board, enterprise leaders emphasize three non-negotiable practices for scaling automation safely:

• Establish Clear Guardrails: Define strict programmatic boundaries for what autonomous agents are permitted to alter autonomously.
• Maintain Human-in-the-Loop Validation: Require explicit human sign-off for any remediation steps involving production database mutations or security access modifications.
• Continuous Feedback Integration: Feed every post-incident review back into the core knowledge graph to prevent recurring failure modes.

Ticket Elimination Before Added Support Headcount

When enterprise ticket volumes surge, the instinctive corporate reaction is often to scale up human headcount by hiring additional Tier 1 and Tier 2 support staff. Walmart’s Dave Glick forcefully contends that this approach fundamentally treats the symptom while ignoring the disease. In enterprise software engineering, engineering leaders are locked in a permanent zero-sum balancing act: every minute spent servicing recurring, low-value operational support tickets is time stolen away from engineering customer-facing features that drive revenue and growth.

+--------------------------------------------------------------------------+
                    ELIMINATE, AUTOMATE, OPTIMIZE
+--------------------------------------------------------------------------+
  1. ELIMINATE:  Remove root causes of paper cuts & recurring tickets.
  2. AUTOMATE:   Deploy agentic AI to handle repetitive L1 workloads.
  3. OPTIMIZE:   Reinvest recovered engineering hours into revenue features.
+--------------------------------------------------------------------------+

As Glick summarizes his core operational philosophy: "My mantra is eliminate, automate, optimize. The best ticket is no ticket. If we get out of a whole class of tickets, we can spend our time building features that affect customers and sellers."

Glick directs relentless organizational focus toward what he terms "paper cuts"—the minor, long-standing operational annoyances that degrade user experience over time. In one prominent instance, e-commerce sellers on Walmart’s platform were continually blocked by unexplained error codes triggered by a conservative image upload size limit. Tracing the issue revealed an arbitrary technical restriction; adjusting the parameter required nothing more than a single line of code, instantly eradicating a persistent source of friction and customer support tickets. Left unaddressed, Glick warns, these minor paper cuts invariably compound into major enterprise liabilities.

To systematically eliminate paper cuts, Glick’s internal programs adhere to three strict operational disciplines:

1. Root-Cause Mandate: Treating recurring support tickets as engineering bugs rather than operational inconveniences.
2. Proactive Edge-Case Testing: Utilizing generative AI tooling—such as automated coding agents—to stress-test new software rollouts against thousands of simulated edge cases prior to deployment.
3. Capacity Reinvestment: Redirecting recovered engineering hours explicitly toward core product expansion rather than absorbing headcount growth.

Demonstrating the raw velocity of modern AI tools, Glick notes that a coding agent recently drafted the entire test suite for a major corporate careers site launch in approximately one week—a monumental task that would have historically consumed months of manual engineering labor. Without the automated agent, unexpected edge cases raised by the product team would have severely jeopardized the hard-coded launch deadline. The true institutional legacy of these deployments is a permanent cultural shift: Glick’s engineering teams no longer wait for production systems to fail before asking how artificial intelligence can optimize the workflow.

Assaf Resnick reinforces this proactive philosophy by positioning true incident prevention—rather than merely accelerating reaction times—as the ultimate objective of agentic IT operations. By analyzing historical telemetry, intelligent change-management agents can review proposed software alterations in real-time, automatically intercepting and halting high-risk deployments before they touch production. In the case of the faulty ATM network update that triggered an eight-hour enterprise crisis, an automated change-governance agent would have flagged the anomalous dependency configuration instantly.

Similarly, Dhrubojyoti Das Deb points to a simple, infallible diagnostic signal for engineering leadership: whenever an incident repeats itself more than once, the repetition is an explicit organizational failure to address the underlying root cause.

Synthesizing the insights shared across the podcast series, enterprise leaders looking to control escalating support costs without inflating headcount must anchor their strategies in three foundational initiatives:

• Audit and Eradicate Recurring Failures: Treat every recurring support ticket as an engineering defect demanding permanent architectural remediation rather than manual operational triage.
• Deploy Predictive Change Governance: Implement autonomous agents to vet code updates and infrastructure modifications against historical failure graphs prior to production deployment.
• Reinvest Recovered Capacity: Ruthlessly channel hours reclaimed through AI automation into feature development and revenue-generating product engineering, ensuring that operational efficiency directly fuels enterprise growth.

Ultimately, as the economics of enterprise technology continue to evolve, organizations that successfully transition from human-intensive firefighting to intelligent, agentic IT operations will secure a decisive competitive advantage—protecting their bottom lines, preserving customer trust, and ensuring robust operational resilience in an increasingly automated world.