Security · IAM · CDK

Replacing PowerUserAccess without breaking the workers

On the old platform, the API and every background worker shared one role with PowerUserAccess. Moving to ECS meant giving each its own role, and then finding out one AccessDenied at a time what they actually called. I replaced the guessing with an audit and a test.

Fortress OS · Sep 2026 · Audit, policy design, and CDK change

• 9 / 26permission checks the workers passed before
• 26 / 26passing after, on all 10 roles
• 48failed re-enqueues in the week before
• 1shared policy, with a test that fails on drift

The problem

On Elastic Beanstalk, the API and four background workers ran under one EC2 role with AWS's PowerUserAccess policy. Nothing was ever denied, so nobody knew what each process really needed.

The ECS migration gave each service its own task role, built from what that service was believed to call. The workers then hit AccessDenied one call at a time. Two gaps had already been patched by hand in the console. Production hadn't moved yet, so every one of these failures was waiting to recur at cutover.

Audit first

• Inventoried every AWS SDK call in the backend and which processes can reach it.
• Turned that into a 26-check matrix: EventBridge, specific queues, buckets, secrets, Lambda invocations, and parameter writes.
• Ran iam simulate-principal-policy against the live roles and searched a week of logs for AccessDenied.

The workers passed 9 of 26 checks. Even the API role was missing several, including invoke rights on two Lambdas behind a feature that was already failing. One worker's re-enqueue onto its own FIFO queue had failed 48 times that week.

The design decision

Strict per-worker least privilege sounds right, but these workers run the same command handlers as the API. Any job can reach any code path the API can. Per-worker policies would just re-create the drift that caused the problem.

So I made one shared runtime policy in CDK and applied it to the API and every worker that runs those handlers, in both the main and ephemeral environments. Workers that don't run the handlers, like the webhook relay, keep narrow, queue-scoped roles. The trade-off, stated in the PR: some workers now hold permissions they may not strictly need. That's still far narrower than PowerUserAccess, and they can no longer fall out of sync with the API.

Make drift a failing test

• Tests assert that the shared policy grants each previously denied call.
• One test requires every worker role to contain every statement the API role has. If someone adds a permission to the API and forgets the workers, CI fails.
• Re-ran the 26-check matrix with simulate-custom-policy against the synthesized policies: 26 of 26 on all 10 roles, with the webhook roles still narrow.
• Checked the patterns against real production resource names before the cutover. That caught one bucket whose prod name didn't match the dev naming pattern.
• Kept the largest inline policy at about 5.2k characters, under IAM's 10,240-character limit.

Outcome

The shared policy replaced the per-worker patches it covered, including the two console-only fixes, which were queued for deletion after deploy. The production cutover inherits a policy that has been simulated and tested instead of one assembled from incident reports.

This write-up describes work at Fortress OS. Details are generalized, and no customer data or internal identifiers are included.