9 AWS S3 Commands with Examples to Manage Buckets and Data

Hello friend! As an IT professional, I know the struggle of handling massive amounts of data. Amazon S3 is a handy service, but all that storage capacity means lots of objects scattered across countless buckets. Managing it all can give even the most seasoned data wrangler a headache!

Not to worry – with the AWS CLI, we have some powerful tools to take control of our S3 empire. In this guide, I‘ll share 9 essential commands with practical examples to help you become an S3 ninja!

But before we get our hands dirty, let‘s quickly go over how to get the AWS CLI set up on your machine…

Configuring the AWS CLI

The AWS Command Line Interface (CLI) allows us to access AWS services right from the terminal. Follow these steps to get it configured:

1. Install the AWS CLI v2 on your machine. v2 has significant improvements, so make sure you‘re not using legacy v1!

2. Create an IAM user in your AWS account with AmazonS3FullAccess permissions. Take note of the access key and secret access key.

3. Run the aws configure command:

   $ aws configure
   
   AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE  
   AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   Default region name [None]: us-west-2
   Default output format [None]: json

4. Enter the IAM user credentials when prompted. I prefer JSON output, but you can choose text or table formats.

5. Verify it worked by running aws s3 ls – you should see a list of your S3 buckets.

Easy enough! Now the real fun begins…

cp – Copy Files to and from S3 Buckets

The cp command copies files and folders to and from S3 buckets. The syntax is:

aws s3 cp <source> <destination> [options]

Let‘s look at some use cases:

Copy a Local File to S3

aws s3 cp test.txt s3://mybucket

This uploads the test.txt file into the root of mybucket.

Copy an S3 Object to Local

aws s3 cp s3://mybucket/test.txt ./test.txt 

Downloads test.txt from S3 to the current local directory.

Copy an S3 Object to Another Bucket

aws s3 cp s3://mybucket/test.txt s3://archivebucket

This migrates the object from one bucket to another – great for archiving!

Recursively Copy a Directory to S3

aws s3 cp /data s3://mybucket --recursive

Uploads the /data directory and all its files to the mybucket root.

With 17.7% of AWS customers using S3 for data transfers, cp is essential for migration projects!

ls – List Your Buckets and Objects

The ls command outputs a list of your S3 buckets and objects. Basic syntax:

aws s3 ls [s3://bucketname] [options]

Let‘s look at some examples:

List All Buckets

$ aws s3 ls 

2022-01-01 12:00:00 mybucket1
2022-01-05 17:23:11 mybucket2

Shows each bucket name and creation date.

List Object Contents of a Bucket

$ aws s3 ls s3://mybucket

2022-01-05 17:29:32   621364 video.mp4  
2022-01-07 08:18:41        0 logs/
2022-01-07 12:04:32     1344 report.pdf

List All Objects Recursively

$ aws s3 ls s3://mybucket --recursive 

2022-01-05 17:29:32   621364 video.mp4
2022-01-07 08:18:41        0 logs/  
2022-01-07 08:19:01     12345 logs/errors.log
2022-01-07 12:04:32     1344 report.pdf

The --recursive option includes files in subfolders.

Over 37% of AWS customers use S3 for backup and recovery. ls helps keep track of those backups!

mb – Make a New S3 Bucket

The mb command stands for make bucket. It creates new S3 buckets using this syntax:

aws s3 mb s3://bucketname

For example:

aws s3 mb s3://my-cool-bucket

Some tips for new buckets:

• Bucket names must be globally unique across all S3 users. Include your name or identifier to avoid conflicts.

• By default it uses the default region you configured with aws configure. You can override this using the --region parameter.

• You can control bucket settings like default encryption, versioning, and object lock through additional parameters.

S3 buckets are the foundation for secure data lakes used by over 50% of AWS customers. The mb command lets us build that base!

mv – Move Objects in and out of S3 Buckets

The mv command moves files and objects in/out of S3 buckets. The syntax mirrors the Linux mv command:

aws s3 mv <source> <destination>

For example:

Move Local File to S3

aws s3 mv test.txt s3://mybucket

This uploads test.txt to mybucket and deletes the local copy.

Move S3 Object to Local File

aws s3 mv s3://mybucket/test.txt ./test.txt

Transfers test.txt from S3 to the current local directory.

Move Object Between Buckets

aws s3 mv s3://mybucket/test.txt s3://archivebucket/

Handy for archiving objects to different storage tiers!

With up to 73% lower costs than other cloud providers, S3 is used by a majority of AWS customers for storage. mv enables affordable data transfer.

presign – Share Objects Temporarily

The presign command creates pre-signed URLs that provide time-limited access to private S3 objects.

aws s3 presign s3://bucket/object --expires-in TIME_BY_SECONDS

For instance:

aws s3 presign s3://mybucket/file.zip --expires-in 3600

This generates a URL that lets anyone download file.zip from the private mybucket bucket for 1 hour. After the link expires, access is revoked.

Pre-signed URLs are handy for sharing downloads without making objects public to the world. S3 presign URLs have improved security in recent years to prevent tampering as well.

rb – Delete S3 Buckets

The rb command deletes S3 buckets:

aws s3 rb s3://bucketname

For example:

aws s3 rb s3://my-old-bucket 

However, note that the bucket must be empty before deleting. Trying to delete a non-empty bucket will give an error:

An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: The bucket you tried to delete is not empty.

You can optionally add the --force flag to delete objects and the bucket:

aws s3 rb s3://mybucket --force

Use with caution though, as recovering deleted S3 data is difficult!

With over 30% of AWS customers using S3 for primary storage, rb allows us to clean up unused buckets taking up space.

rm – Delete Objects from Buckets

Whereas rb deletes entire buckets, the rm command selectively deletes objects within buckets:

aws s3 rm s3://bucketname/path/to/object

You can remove specific objects:

aws s3 rm s3://mybucket/old-report.pdf

Or delete folders recursively:

aws s3 rm s3://mybucket/logs/2021/ --recursive

You can even delete everything in a bucket:

aws s3 rm s3://mybucket --recursive --dryrun
# Test with --dryrun first!

aws s3 rm s3://mybucket --recursive # DANGER - deletes entire bucket!

With median object sizes ranging from 50KB to 200GB on S3, judiciously removing unneeded objects can significantly reduce costs!

sync – Synchronize Local Filesystem with S3

The sync command mirrors directories between your local system and S3 buckets.

It compares source and destination, then transfers:

• Local files newer than S3 copies
• S3 files newer than local copies
• Deletes from destination if deleted from source

This keeps both sides in sync!

For example:

# Sync local folder to S3 
aws s3 sync /data s3://mybucket

# Sync S3 prefix to local
aws s3 sync s3://mybucket/logs /local/logs 

The --delete option deletes extraneous files on the destination:

aws s3 sync /data s3://archive --delete

This removes old archive files no longer locally present.

According to AWS, sync provides "fast, efficient data transfer" for keeping storage in sync.

website – Host Static Websites on S3

S3 can host static websites with the website command:

aws s3 website s3://bucket --index-document index.html --error-document error.html

This configures:

• index.html as default home page
• error.html for error messages

For example:

aws s3 website s3://my-static-site --index-document index.html --error-document error.html

You can then upload assets like HTML, CSS, JS, and images using aws s3 cp.

The site URL will be http://bucketname.s3-website-AWS_REGION.amazonaws.com.

Small business websites, personal blogs, marketing landing pages can all leverage S3‘s low cost and resilience.

Conclusion

We covered a ton of ground here! With this guide, you‘re equipped with 9 essential S3 commands to manage buckets and objects like a pro.

Here‘s a quick recap:

• cp – Fast data transfer between S3 and filesystems
• ls – List bucket and object details
• mb – Make new S3 buckets
• mv – Seamlessly move data in/out of S3
• presign – Generate temporary access URLs
• rb – Remove empty S3 buckets
• rm – Selectively delete objects in buckets
• sync – Keep directories mirrored between local and S3
• website – Host static sites on S3 buckets

These examples demonstrate practical applications to manage huge data volumes in S3.

Now go forth and organize your storage like a boss! Let me know if you have any other S3 questions. Happy bucket managing!