Periodically download backups of Google App Engine Datastore

Recently I came across another interesting issue. I hope that this post will be helpful for all developers who develop their software as a service applications using Google App Engine.


After creating an application I wanted to schedule Google Datastore backups to be done every evening. Apparently it was not that strait forward. There is a tool called Gaebar (http://aralbalkan.com) but it works only for Django projects.


Google provides mechanism for downloading and uploading data:


http://code.google.com/appengine/docs/python/tools/uploadingdata.html


however I couldn't find a solution that I could use to schedule downloads. 
Below is a solution for Windows.


I have created a batch file for downloading a datastore backup:
backupGAE.bat source:


@ECHO OFF
SET EMAIL=%1
SET PASS=%2
SET APPID=%3
SET OUTPUT_FILE=%4


echo %PASS% | "C:\Python25\Python.exe" "c:\Program Files\Google\google_appengine\appcfg.py" download_data --email=%EMAIL% --passin --application=%APPID% --url=http://%APPID%.appspot.com/remote_api --filename=%OUTPUT_FILE%


Explanation:
--passin
"If given, the tool accepts the Google account password in stdin instead of prompting for it interactively. This allows you to invoke the tool from a script without putting your password on the command line."


"|" symbol causes that the output of the first program is sent to the second program.

Then I wrote an application calls the backupGAE.bat file with required parameters:
-email address
-password
-application id
-output file


Then compresses the downloaded file and secures it with password and deletes the original file.
Because I couldn't find any freeware python zip library that allows setting passwords on new archives I have used a 7-zip command line - works very well.


The application execution is scheduled on the server using Microsoft Scheduled Tasks and after is executed send an email status to a given email address.