We removed our free Sandbox April 25th.
You can read more on our blog.

Python worker

Note

CLI Command examples on this page are always provided without the --application (shorthand -A) argument, assuming you’re running these commands in a connected folder (at creation or using the dotcloud connect command). For more details on connected folders, see Migrating to the CLI 0.9.

Sometimes, you need to run a program for a longer time than a single HTTP request. Use cases include:

  • CPU-intensive jobs (e.g. video transcoding);
  • running some code at a specified time, or on regular intervals;
  • background activity (e.g. crawling 3rd party service to update your database);
  • run a specific web server, like Node.js or Tornado;
  • and much more!

dotCloud provides “worker services” dedicated to those tasks. There is a different service for each language: Ruby Worker, PHP & PHP Worker, Perl Worker, Python worker, Node.js... The only difference between them is the set of pre-installed packages, and dependencies handling: python-worker supports requirements.txt, while —for instance— ruby-worker supports Gemfile.

All worker services rely on Supervisor to start and monitor your processes. Supervisor will start defined programs automatically, and will restart them automatically if they crash or exit. If you just need to run a program at a specified interval, you can also use a crontab and ignore Supervisor.

Note

You can also use a “non-worker” service to run some background jobs. More specifically, all services feature crontab, allowing you to run Periodic Tasks. So if you want to run a daily Python script, e.g., a stock ticker in your database, you don’t have to dedicate a python-worker to this task: you cansetup the crontab in the same python service that you use for your web application. However, you should be aware that when you scale your application, the cron tasks will be scheduled in all scaled instances – which is probably not what you need! So in many cases, it will still be better to use a separate service.

Similarly, a lot of (non-worker) services already run Supervisor, so you can run additional background jobs in those services. Then again, remember that those background jobs will run in multiple instances if you scale your application. Moreover, if you add background jobs to your web service, it will get fewer resources to serve pages, and your performance will take a significant hit.

Technically, if you really want to know – there is almost no difference between worker and non-worker services. For instance, the python-worker service is basically the python service without Nginx (HTTP server) and uWSGI (Python web workers) running. Both can optionally run background processes using Supervisor.

Basic Use

Let’s assume that you are building a dotCloud application called “ramen”. For the sake of simplicity, we will also assume that everything related to this application is located in a directory called “ramen-on-dotcloud”.

Let’s setup our environment:

mkdir ramen-on-dotcloud
cd ramen-on-dotcloud
dotcloud create ramen

A dotCloud application is described by a Build File (dotcloud.yml), which is a simple YAML file named “dotcloud.yml” located in our “ramen-on-dotcloud” directory. To add a new service to our app, we just add the following lines to “ramen-on-dotcloud/dotcloud.yml”:

worker:
  type: python-worker

With your dotCloud build file in hands you are ready to define your daemons.

Specify Python dependencies

Just like for the Python service, you can specify external dependencies in a file called requirements.txt. For example:

redis
restkit == 2.3.3

If you specify a Python package installing executable scripts or binaries, they will be correctly installed in the $PATH and you will be able to run them without specifying their full path.

Python Versions

The python-worker service supports four different branches of Python (2.6, 2.7, 3.1, 3.2), it will default to Python 2.6 unless you specify otherwise. The current versions of each Python branch are listed in the table below. Pick the branch that works best for you.

branch version variable
2.6* 2.6.5 v2.6
2.7 2.7.2 v2.7
3.1 3.1.2 v3.1
3.2 3.2.2 v3.2

* Python 2.6 is the default

To configure your service to use a version of Python other than 2.6, you will set the python_version config option in your dotcloud.yml.

For example, this is what you would have if you wanted Python 2.7.x:

www:
  type: python-worker
  config:
    python_version: v2.7