It’s 2024, and AWS finally built a browser-based S3 viewer. Twenty years after S3 launched, you can now browse your buckets directly in the browser. It’s still in alpha, but hey, better late than never, right?

You might be thinking - wait, weren’t there options before? Like S3Fox? Yeah, there were some third-party tools, but AWS itself never had an official browser interface. So what took so long?

Real Problem: Access Control

Turns out, building a simple file browser wasn’t the hard part. The hard part was making it work with proper access control. AWS had to basically reinvent how S3 access works. Twice.

Why? Two letters: AI. Specifically, data lakes. When you’re dealing with massive amounts of data spread across multiple buckets and accounts, the old IAM roles and bucket policies just don’t cut it. They’re too limited and honestly kind of clunky at scale.

Enter S3 Access Grants, announced at re:Invent 2023. This is the new recommended way to provide access to S3 data (which, let’s be honest, means everything else is now “not recommended”).

Evolution of S3 Access

We’ve gone through quite a journey:

  • S3 ACLs (the old way)
  • S3 Bucket Policies
  • IAM policies
  • S3 Access Points
  • S3 Access Grants (the new hotness)

With Access Grants, you create a grant that lets a user access a specific bucket or prefix. The user authenticates through something like Okta or Microsoft Entra ID via IAM Identity Center, and gets centralized access to their data across accounts and buckets - even from outside AWS.

Listing Problem

But here’s where it got tricky. You could create all these grants, but you couldn’t actually list what a user had access to across all buckets and prefixes. It’s the same problem we’ve always had with S3 - you can’t limit bucket listing to only show buckets a user can access.

Remember that eternal pain point? s3:ListAllMyBuckets requires a wildcard in resources, so it shows everything. There was no way to show only the buckets a user actually has access to.

The Solution: ListCallerAccessGrants

Finally, AWS introduced s3:ListCallerAccessGrants. This API call lets you get a complete list of S3 buckets and prefixes that are accessible to the current user through their Access Grants. No more showing them every bucket in the account - just what they can actually access.

This is what made the Storage Browser for S3 possible. Without it, you’d still be stuck showing users everything or nothing.

Why This Matters

This might seem like a small thing, but it’s actually huge. For years, we’ve been working around S3’s access limitations. Now we have a proper way to build user-facing data applications that can show users exactly what they have access to, nothing more, nothing less.

The Storage Browser is cool, but the real win here is s3:ListCallerAccessGrants and the S3 Access Grants system. It’s the foundation that makes proper multi-tenant data applications possible.

Sometimes the simplest features take the longest to build because they require rethinking everything that came before.

References: