Loading

Service Disruption on May 21-22, 2026

Publish Date: Jun 1, 2026
Description

Root Cause Analysis 

Published on June 01, 2026

We sincerely apologize for any impact this incident may have caused you and your business. At Salesforce, Trust is our #1 value, and security is our top priority. We value transparency and want to take this opportunity to outline the facts regarding a recent service disruption that may have affected your ability to use multiple Salesforce services.

Executive Summary - What Happened?

On May 21, 2026, and May 22, 2026, customers experienced login failures and service access issues with Salesforce services via login.salesforce.com and test.salesforce.com. The disruption was caused by disk space exhaustion on the Public Proxy (Egress Gateway) infrastructure, which mediates outbound traffic from Salesforce applications to public destinations. A defect in the log rotation mechanism, triggered by a configuration change in the underlying infrastructure software, caused log files to accumulate unboundedly, eventually exhausting disk space on proxy hosts in multiple data centers. The Salesforce Technology team (Technology Team) resolved the issue by clearing the accumulated logs and reloading the hosts. Subsequently, a permanent fix has also been deployed.

How did this issue impact Salesforce services?

During the impact window, users could not access their Salesforce services. Some customers who used login.salesforce.com may have experienced disruption while accessing their Salesforce application. Access using my.salesforce.com was working as expected.

The following orgs were impacted: cs238, cs240, cs244, cs245, cs249, na224, na226, cs254, cs257, cs259, cs260, cs261, cs264, cs265, cs268, cs269, cs270, cs271, cs340, cs342, cs344, cs272, cs273, cs274, cs275, cs276, cs277, cs278, cs280, cs281, cs282, na231, na235, na241, na245, na247, na249, na251, na252, na254, cs236, cs237, cs239, cs241, cs242, cs243, cs246, cs247, cs248, cs250, cs251, cs252, cs253, na223, na225, cs255, cs256, cs258, cs262, cs263, cs266, cs267, cs341, cs343, cs345, cs279, cs283, na232, na233, na234, na236, na237, na238, na239, na240, na242, na243, na244, na246, na248, na250, and na253.

Technical Details

Detection and Initial Impact:

On May 21, 2026, at 20:00 Coordinated Universal Time (UTC), the Salesforce infrastructure monitoring detected degraded availability on the Public Proxy fleet in the IA6 and IA7 data centers. Multiple proxy hosts became unresponsive after their root filesystems exceeded critical disk utilization thresholds.

On May 21, 2026, at 21:00 UTC, the Technology team engaged and launched an investigation.

Remediation:

On May 22, 2026, at 01:37 UTC, the Technology Team cleared the accumulated log files and restarted the affected proxy services.

On May 22, 2026, at approximately 18:00 UTC, the Technology Team deployed a permanent fix via the standard deployment pipeline, correcting the log rotation configuration across the entire proxy fleet.

On May 23, 2026 through May 25, 2026, the Technology Team verified the fix across all data centers, confirming successful log rotation cycles and healthy disk utilization levels.

Root Cause Analysis

The Technology Team's post-incident investigation identified the root cause as a configuration defect in the Public Proxy service's log rotation system. The configuration referenced a system group, a preconfigured user group that no longer exists in the infrastructure. As a result, log rotation failed to execute, causing logs to accumulate without being cleaned up. Eventually, accumulated logs filled the server's storage, preventing the Public Proxy service from processing outbound traffic.

The Technology team identified a monitoring gap as a contributing factor to the prolonged impact period. The monitoring threshold for the disk usage did not detect the accumulated logs. The Technology team is enhancing the monitoring and expanding the alerting thresholds, so similar issues can be detected more quickly in the future. 

 

Next Steps

To maintain the performance level that our customers expect from Salesforce and to prevent this from recurring, our focus is on continuous improvement. The Technology Team has identified and is implementing the following actions:

Completed:

  • The Technology Team removed the invalid system group reference from the log rotation component of Public Proxy software. This fix has been deployed and verified across all data centers.

In Progress:

  • The Technology team is upgrading alerting thresholds so that high disk usage over a certain threshold triggers immediate high-priority notifications, enabling faster detection and response to storage-related issues before they impact service availability.

  • The Technology team is assessing the feasibility of relocating log files from critical system partitions to dedicated storage locations, further reducing the risk of disk exhaustion impacting core services.

 

We sincerely apologize for the impact this incident may have caused you and your business; Salesforce is fully committed to minimizing downtime when incidents do occur. We also continually assess and improve our tools, processes, and architecture to provide you with the best service possible. 

 

Knowledge Article Number

005385903

 
Loading
Salesforce Help | Article