onemusicapi.com uses S3 to cache images that are loaded from upstream data sources and sent to clients. The bucket used for these images allowed anonymous upload, but this was never a security issue because files were only uploaded from EC2 instances that had a suitable EC2 IAM role.
We created an EC2 instance that didn’t have any EC2 IAM roles, and used it to add some images to the cache. After the process was complete (which took several days), we found that the added images were not accessible, and found that it was due to permissions errors from S3. Further investigation in the AWS console confirmed this, and also showed that the object ACLs weren’t accessible.
Based on an AWS forum post, it turned out that the objects had been uploaded as the “anonymous” user (this was allowed in the bucket ACL). This meant that requests made as an authenticated user were denied access to objects; they were supposed to be accessed as the anonymous user only, even though it is trivial to switch to the anonymous user (using --no-sign-request in the AWS CLI or AnonymousCredentialsProvider.create() in the Java SDK). This is a counter-intuitive aspect of S3 permissions behaviour.
Once we realized this, we were able to follow these instructions for changing the ownership of an object to verify that this was indeed the problem:
$ aws s3api put-object-acl --bucket BUCKET-NAME --key OBJECT-KEY --acl bucket-owner-full-control --no-sign-request
$ aws s3 cp s3://BUCKET-NAME/OBJECT-KEY s3://BUCKET-NAME/OBJECT-KEY --storage-class STANDARD
Knowing the problem and how to fix it, we had to apply the fix to around 1.1 million objects in the bucket, while avoiding updating the objects that already had the correct owner and permissions, because updating all the objects would have incurred a higher S3 cost.
We found a couple of scripts on GitHub, but they all updated all the objects in the bucket, and there was no way to filter by prefix or ownership (we needed to filter by both). So we wrote a tool using fs2-blobstore to filter the relevant objects and update the ownership and permissions.
The tool is available on GitLab.
By Waqqas Dadabhoy
Photo by Chris Yang on Unsplash