What I’ve learned

How I Locked the Whole Company out of an Amazon S3 Bucket

An S3 policy anti-pattern to avoid; And how to unlock an S3 bucket

Carina Nebula by NASA, ESA and STScI

Recently, I did a review of security policies for one of my customers. In the process, I managed to accidentally lock myself and everyone in the company, out of an S3 bucket. It was not just any bucket, no — it was the bucket holding all the customers' media files. Imagine having to ask all your customers to reupload all their photos, videos etc…

An anti-pattern S3 policy to avoid

Let’s see what I did so wrong, when I attached the following policy to the S3 bucket.

Anti-pattern S3 policy statement that will render a bucket fully locked.

Attaching the above policy to an S3 bucket will lock everyone out of that bucket. Before we get into the details, let me tell you two important things first:

• Try avoiding NotPrincipal, if at all possible. If you decide use, it will probably come back and bite you when you least expect!
• There is a way to fix it. It is actually quite easy, BUT you will need the root access keys. At the end of the article, I show what needs to be done to fix the locked bucket.

Let’s now take a closer look at the above S3 policy and why it is problematic.

A NotPrincipal anti-pattern

In short, the example policy is intended to block access to the data (objects), but also the bucket itself to anyone except a user or service who has the role ROLE_NAME assumed i.e., attached to it. Unfortunately, our example policy does not specify this intension correctly and even assuming the role will not grant a user with the access.

At the first glance it seems that if you try to access the bucket with an IAM user, after they have assumed the role, the access would be granted. Well, it will not. The action (whatever it is) will be denied. Even if the user has an Admin access or a policy with S3FullAccess, they will be denied.

The problem lies in the “inverted logic” i.e., double negation approach, which in this case ends up always being true. The NotPrincipal proves problematic with IAM roles because the role’s Principal ARN value slightly changes when the role is assumed by an user or a service. In particular, AWS Security Token Service (STS) appends a session ID to the Roles ARN. Look at the outputs of the following two commands:

$ aws sts get-caller-identity
{
  "UserId": "UNIQUE_ROLE_ID:SESSIONID"
  "Arn": "arn:aws:sts::111111111111:assumed-role/ROLE_NAME/SESSID"
}
$ aws iam get-role --role-name ROLE_NAME
{
  "RoleName": "ROLE_NAME",
  "RoleId": "UNIQUE_ROLE_ID",
  "Arn": "arn:aws:iam::111111111111:role/ROLE_NAME",
  "AssumeRolePolicyDocument": {...}
}

We can see that the role’s ARN is different than the ARN of an IAM caller (user or service), making the request to the bucket. The role ARN is the identifier for the IAM role itself and the assumed-role ARN is what identifies the role session in the logs. One might be tempted to uses a wildcard in the NotPrincipal. However, this is not allowed and for a good reason.

So what I should have done in the first place is simply not use NotPrincipal. Instead, I should have used the Condition with a aws:userId, as shown in the following snippet.

Best practice S3 policy using aws:userId instead of NotPrincipal.

It is important to notice that in our condition we are not using role’s ARN, but rather its unique Role Id to specify the desired role. To retrieve the Role Id, we can run $ aws iam get-role --role-name ROLE_NAME. One can easily identify role’s unique ID, as in AWS all role unique IDs are prefixed with AROA. A keen-eyed reader will have also noticed that we had to suffix our role’s unique ID with:* . The reason is that the STS will append a session ID also to the role’s ID, as shown in our example outputs above.

How to fix it

While it is true that, under the aforementioned policy, no user or service would be allowed to perform any action on the bucket or its objects, there is an exception. Yes, it is the root account, which is always allowed to view and delete S3 security policy.

The only reliable way to unlock the inaccessible bucket is to actually delete the problematic S3 security policy. To do that, simply run the following command:

$ aws s3api delete-bucket-policy --bucket BUCKET --profile ROOT

The --profile argument can be omitted if you are managing credentials via environment variables. To add the credentials to the env, one can run:

$ export AWS_DEFAULT_REGION= us-east-1 
$ export AWS_ACCESS_KEY_ID=ROOT_KEY_ID
$ export AWS_SECRET_ACCESS_KEY=ROOT_SECRET_KEY

Closing Thoughts

In this article, we looked at an anti-pattern for writing S3 security policies and issues it can cause. We have also seen how to fix the issues, if one has access to the root account credentials. Nevertheless, we should clearly avoid writing bad S3 policies in the first place. Here are some (very general and succinct) best practices when writing S3 policies:

• Try avoiding generalized wildcards such as “*” or “s3:*” in Action and Principle, without reducing the statement’s scope a with Condition.
• Try avoiding NotPrincipal. It can almost always be replaced with a Condition and aws:userId or aws:PrincipalArn.
• Try using IAM Access Analyzer. It is far from perfect, but it can be quite useful in some situations. Especially if one has to deal with many policies and resources.