Reliability · ECS · SQS

Deploys that don't stall the queue

Moving our background workers from Elastic Beanstalk to ECS uncovered two gaps. Deploys were freezing customers' financial queues, and shutdown signals never reached the process. Here's how I fixed both.

Fortress OS · Oct 2026 · Designed and shipped

• 30 minqueue stall per interrupted message on the old path
• 0 of 193worker starts that ever logged a graceful shutdown
• 226duplicate redeliveries in one week on the new path
• 4workers moved to one shared, tested consumer

Context

Four Node.js workers handle financial, reporting, communications, and general jobs from SQS FIFO queues, with one message group per customer organization. FIFO keeps each customer's financial messages in order. It also means one stuck message blocks everything behind it for that customer.

Three problems, one root

• On Beanstalk, a deploy could stall a customer for 30 minutes. The platform's queue daemon used a 1,800-second visibility timeout. When a deploy interrupted a long financial message, that customer's queue waited out the full timeout. CloudWatch showed this on four production deploys in one month.
• On ECS, the opposite. The workers set no lease at all, so they inherited 30-second queue defaults. Long jobs became visible again while still running. In one week, a single dev worker logged 67 expired-receipt delete failures and 226 duplicate redeliveries.
• SIGTERM never reached Node. The container start scripts ran npm run, npm runs scripts under sh -c, and dash dies on SIGTERM without forwarding it. The existing graceful-shutdown code had never run once: zero shutdown requested lines across 193 starts.

Leases sized from data, not guesses

Each worker now holds a short visibility lease on its current message and renews it with ChangeMessageVisibility while the handler runs. A message only reappears if a full lease passes with no renewal, which means the worker died or its event loop stalled.

That makes the lease length an event-loop question. I set each lease from the 30-day maximum of event_loop.delay.max on the production workers:

A bounded drain

• The start scripts now exec node directly, so the init process delivers SIGTERM straight to Node.
• On SIGTERM, work that finishes within 25 seconds completes and is deleted normally.
• Anything still running at the deadline is released with visibility 0. The replacement task picks it up immediately instead of waiting out the lease. All of this fits inside ECS's 30-second stop timeout.
• Puppeteer's own signal handlers came out. Once SIGTERM actually reached Node, they would have killed Chrome mid-render during the drain and dropped the letter being generated.
• The four workers' duplicated consumer wiring became one shared module with specs. It logs every consumer error with the message, group, and receive count.

Proving it

• Ran a container test on the real image. With the old script, docker stop took 0.06s, exited 143, and the shutdown handler never ran. With the new one, the drain ran, a simulated 5-second job finished, and the container exited 0 after 5.06s.
• Ran the real consumer library against a fake SQS client for every worker's lease. The specs cover renewals during long handlers, the retry delay after a throw, and release at the deadline.
• Wrote down the known limits next to the code. A hung handler still holds its group. Some side effects outside the message's transaction can repeat on redelivery, which now comes within one lease instead of 30 minutes. The infrastructure follow-ups (stop timeouts, container memory, runtime metrics) went on the list with them.

Outcome

The change is merged and running on our ECS environments. Production picks it up at the ECS cutover, so the leases and the drain arrive with the migration rather than after a bad deploy proves they were needed.

This write-up describes work at Fortress OS. Details are generalized, and no customer data or internal identifiers are included.