A client’s site returned intermittent HTTP 503s that were impossible to reproduce. The trail led to a Lambda@Edge origin-response function hitting a 3-second timeout on sequential cross-region calls to DynamoDB in us-east-1 and S3 in eu-west-1 — while executing in Asia Pacific. Here’s how CloudFront’s request path explains it.
Our ECS worker was healthy, not crashing, not stuck — yet autoscaling killed it right as it picked up a new job. The fix? Inverting the ECS Task Protection lifecycle so the worker is protected whenever it’s ready to accept work, not just while processing.
We reduced EFS costs by 70% by moving temporary file operations to ephemeral storage. Everything looked perfect — until production started acting strange. Here’s what went wrong, why it happened, and how we built a proper decoupled architecture following AWS Well-Architected principles.