In this tutorial, we shall learn to move a file from an SFTP server to an S3 server on the Amazon Web Services (AWS) cloud storage service. Here, while talking of AWS S3 upload, we will be dealing with two servers, one will be an SFTP server from where we will copy the files, and then we will try to upload that file into the Amazon S3 server.
To practice what you learn in this tutorial, you must have an active account on AWS with an enabled S3 service (you can follow the instructions given on the official page of AWS to set up the account and activate the S3 service. You will find the link to it in the “References” section at the end of this post). You must have access to the SFTP server, “test.rebex.net”.
To begin, we need to install Boto3, a Python software development kit (SDK) provided by AWS, and pysftp to communicate with the sftp server.
pip install boto3
pip install pysftp
Download File From SFTP Server To Local Directory
We will use the following program to download the targeted file from our targeted SFTP server, test.rebex.net. If you want to investigate this code further, please refer to this tutorial, where we’ve discussed every thing in detail. We will download, “ResumableTransfer.png” from pub/example folder on the test.rebex.net server.
import pysftp
host = 'test.rebex.net'
port = 22
username = 'demo'
password= 'password'
try:
conn = pysftp.Connection(host=host,port=port,username=username, password=password)
print("connection established successfully")
except:
print('failed to establish connection to targeted server')
conn.get('/pub/example/ResumableTransfer.png')
print('file downloaded to local machine successfully')
Upload To AWS S3 Server
So far we have downloaded the file from an SFTP server to a local directory. Now, we will upload this same file to the S3 server using the same code we used in our last tutorial – How To Upload, Download And Delete File From S3 server.
Now, we import the Boto3 library provided by AWS, and then we create client object named as s3_client.
import boto3
# pass s3 as service name generate s3 client
s3_client = boto3.client('s3')Now pass the name of local file (we want to upload on the S3 server), the name of the bucket created on the S3 server, and the object name that will be used to keep the file on the server. Remember, if you want to upload a file with the same name, then keep the object name the same as the file name.
import boto3
# pass s3 as service name generate s3 cleint
s3_client = boto3.client('s3')
file_name = 'ResumableTransfer.png'
bucket_name = 'datacourses-007'
object_name= file_name
try:
s3_client.upload_file(file_name,bucket_name,object_name)
print('file: ',file_name,' uplaoded on bucker: ',bucket_name,' successfully')
except Exception as e:
print(e)
If we check our ‘datacourses-007’ bucket on the S3 server, we can see that the file is available there as shown below:
Summary
In this tutorial, we learned to move a file from an SFTP to a S3 server. This is a two-step process: first, download the file from the SFTP server (we downloaded a png file from the SFTP server, test.rebex.net). Then, upload the local file to an S3 server, which we did in this tutorial. We uploaded that file to our bucket, datacourses-007 on the S3 server.