What Happened

On October 19-20, 2025, Amazon Web Services experienced a critical infrastructure failure in the US-EAST-1 (Northern Virginia) region lasting approximately 14.5 hours. The incident originated from a race condition in DynamoDB's DNS management system, cascading across 113 AWS services and impacting thousands of companies globally.

As bem is primarily hosted on AWS (utilizing Route53, EC2, and Amazon Textract for various service components) there was significant service degradation and a partial outage stemming from the

Key Findings:

• Root cause: Race condition in DynamoDB DNS automation resulting in empty DNS records, leading to cascading failures affecting bem’s platform
• Duration: 14 hours 32 minutes (11:48 PM PDT Oct 19 - 2:20 PM PDT Oct 20)/

Timeline

• 1:30am PDT
  • DNS outage with app.bem.ai and api.bem.ai unreachable by many clients with inconsistent DNS resolution
  • AWS Textract time out issues begin
• 2:25am PDT
  • DNS recovers
  • Textract timeout issues continue, leading to degraded performance on some inputs processed through bem
• 5:00am PDT
  • Textract issues recover
• 10:40 am PDT
  • Textract timeout issues emerge again due to AWS team’s updates to the AWS Network Load Balancer (NLB)
• 12:40pm PDT
  • Textract issues recover after AWS team’s successful fixes shipped to smooth over the NLB

How We’re Moving Forward

We sincerely apologize for the disruption this incident caused to your workflow. Providing mission critical infrastructure, and the reliability to match that standard, is a core part of our identity. We are taking the following actions to mitigate risks in widespread cloud provider outages like this:

Maturing our customer-facing status monitoring We have implemented a more advanced status page supporting Slack, email, and RSS subscription feeds. The reporting itself will be linked to our own application monitoring as well as the status of our critical infrastructure providers.

Owning more of our vision infrastructure We use a mixture of traditional OCR models as well as newer generation VLMs. Our occasional reliance on Textract as a traditional OCR provider led to degraded performance when our vision classification layer routed to it as a service. We are continuing to mature our usage of VLMs and lessening our reliance on traditional OCR providers.

Multi-region and multi-cloud load balancing To ensure stronger resiliency, we will be implementing multi-region deployments with automated load balancing through the bem API to mitigate risks with single region failures. Soon thereafter, we will implement the same across cloud providers (GCP and Azure in addition to AWS) to ensure additional resiliency with widespread outages across public and private clouds.

We are committed to providing a reliable and trustworthy platform that exceeds industry standards. Thank you for your patience and understanding, and we deeply appreciate your partnership.

Sincerely, The bem Team