Data Courses

In this tutorial, we are going to learn to move a file from an SFTP server to a Google Cloud Storage (GCS) server in Python. We will be using the pysftp library to communicate and fetch data from the SFTP server, “test.rebex.net”. To upload that file to the GCS server we will be using Google Cloud SDK (gcloud SDK) provided by the Google Cloud Platform. So get ready to learn various aspects about Google cloud storage server file transfer.

To proceed, make sure you have the access credentials for the targeted SFTP server. You will also have to activate a Google cloud account with GCS as an active service on it.

Requirements

Install gcloud and pysftp using pip in the following way:

!pip install pysftp
!pip install gcloud

Downloading File from SFTP Server

We will download a png file from /pub/example repo on our SFTP server, test.rebex.net. We will be using a Python program to download the file.

First, import the pysftp library, then, provide the credentials and the server address. After that, create the SFTP connection to the server, and then send the “downloading file” request.
To further explore code in detail please refer to our tutorial of SFTP series.

import pysftp
host = 'test.rebex.net'
port = 22
username = 'demo'
password= 'password'
try:
  conn = pysftp.Connection(host=host,port=port,username=username, password=password)
  print("connection established successfully")
except:
  print('failed to establish connection to targeted server')
conn.get('/pub/example/ResumableTransfer.png')
print('file downloaded to local machine successfully')

So now that we have downloaded the file to a local directory, let us upload this file to the GCS server.

Upload File To GCS

To upload the file to the GCS server, first, we should establish a connection with that server. We have already discussed the details to establishing a connection with a GCS server in a previous tutorial.

Let’s go over to the Python code now. First, consider all storage from gcloud in Python code.

from gcloud import storage

Now, let’s create a storage client as following:

storage_client = storage.Client()

Once the client is created, we need to define variables determining the file name we want to upload on GCS server, the bucket name (where file will be uploaded), and the destination name of file (name of file inside bucket). Remember, in the GCS server files are named as blobs.

file_name = 'ResumableTransfer.png'
bucket_name = 'datacourses-007'
blob_name = file_name

Now, we create a bucket object using storage_client object.

bucket = storage_client.bucket(bucket_name)

Then we create blob by calling blob() method of bucket object

blob = bucket.blob(blob_name)

We can now call upload_from_filename() method to upload file to destination bucket as following:

try:
    blob.upload_from_filename(file_name)
    print('file: ',file_name,' uploaded to bucket: ',bucket_name,' successfully')
except Exception as e:
    print(e)

So our program has printed that the file was uploaded to the bucket successfully. Let’s verify this on the dashboard.

We can see in the above image that “ResumableTransfer.png” file has been uploaded to the GCS server’s bucket “datacourses-007” successfully.

Summary

In this tutorial related to Google cloud server file transfer, we learned to move a file from the SFTP server to the GCS server. We used the pysft library in Python to download the file from test.rebex.net. We uploaded it to the GCS server using gcloud sdk provided by Google. We also learned about establishing connection with the SFTP server and the GCS server, and also explored buckets in the GCS server.

---

References

https://cloud.google.com/docs

https://pysftp.readthedocs.io/en/release_0.2.9/