In this tutorial, we will learn to process files in Python on S3 server (storage service provided by AWS).
Amazon Web Services (AWS) is one of the leading cloud services providers today. S3 is an object cloud storage service provided by AWS. AWS provides an API to interact with S3 in Python. We will learn how to use that API to not only upload files on the S3 server, but also to download and delete them.
Requirements
For this tutorial you must have working Python 3, Boto3 and AWS accounts with an active S3 service on it. To learn about how to create the AWS account and activate the S3 service, please refer to this site.
The Python API provided by AWS is named as Boto3. First, we will have to install Boto3 on our machine. We will be using pip to do so in the following manner:
pip install boto3
Creating Buckets in S3
An Amazon S3 bucket is a cloud storage resource available in AWS S3 to the public. The latter is like file folders, and stores objects consisting of data and metadata. You can create an S3 bucket using the dashboard provided in your account. When you activate the S3 service, it provides you a dashboard that you can use to create the buckets. To read about buckets, click here.
Once you have created a bucket in S3 we can move forward. For this tutorial we have created a bucket, “datacourses-007″as shown in the image below.
To create a new bucket, you can click on the ‘Create Bucket Button”, and can set the name and the other required information easily.
So far we have installed Boto3 and created a bucket on S3. Now, let’s move forward to our Python program to upload the file on to the S3 server.
Uploading Files To S3
To begin with, let us import the Boto3 library in the Python program. Then, let us create the S3 client object in our program using the boto3.Client() method.
import boto3
# create client object
s3_client = boto3.client('s3')Now, pass the file path we want to upload on the S3 server.
import boto3
# create client object
s3_client = boto3.client('s3')
file_name = 'testing.txt'
bucket_name = 'datacourses-007'Now, we call s3_client.upload_file() by passing path of file and bucket name as following.
import boto3
# create client object
s3_client = boto3.client('s3')
file_name = 'testing.txt'
bucket_name = 'datacourses-007'
response = s3_client.upload_file(file_name,bucket_name,file_name)
print('file ',file_name,'uploaded on bucket',bucket_name,'successfully')
So after our code prints that the file has been uploaded successfully, let us check our dashboard on bucket datacourses-007. You can see in the screenshot below that “testing.txt” file is now available in our bucket.
Downloading Files From S3
To download files from the S3 server, we will have to call s3_client.download_file() passing object name and bucket name as parameter.
Remember, a file uploaded on the S3 server is treated as an object, so now our targeted file on S3 server is an object. s3_client.download_file() expects a bucket name, object name (name of file on s3 server) and file name (name that we want to assign to that object file on our local directory).
So for this case our object file is “testing.txt” and we want to download it as, “downloaded.txt” file.
import boto3
# create client object
s3_client = boto3.client('s3')
object_name = 'testing.txt'
bucket_name = 'datacourses-007'
file_name = 'downloaded.txt'
response = s3_client.download_file(bucket_name,object_name,file_name)
print('object 'object_name,'downloaded from bucket',bucket_name,' as ',file_name,'successfully')
Deleting Files From S3
Deleting file from the S3 server using Boto3 is a bit easier as compared to the uploading and the downloading of files.
To do it you must have aws_access_key_id and aws_secret_access_key. By passing those keys, the authentication process takes place successfully.
Let’s start by importing and creating Session object using “boto3.Session”, and we will have to pass keys while generating session.
Then we create Resource object using that Session object.
After that, we create Bucket object by calling s3_resource.Bucket() method.
Once done, we call my_bucket.delete_objects() and we will pass key values pair object as “Delete parameter” to specify which file we want to delete.
import boto3
from boto3.session import Session
aws_access_key='enter your access key id'
aws_secret_access_key='enter your secret access key'
session = Session(aws_access_key_id=aws_access_key,
aws_secret_access_key=aws_secret_access_key)
s3_resource = session.resource('s3')
my_bucket = s3_resource.Bucket("datacourses-007")
response = my_bucket.delete_objects(
Delete={
'Objects': [
{
'Key': "testing.txt" # the_name of_your_file
}
]
}
)So after we run this code we can verify that testing.txt has been deleted from our bucket successfully. See below image.
Summary
In this tutorial we learned to use AWS provided Boto3 and a Python API to develop AWS applications in Python. We explored methods to upload, download and delete files in the S3 server using client program written in Python.