We removed our free Sandbox April 25th.
You can read more on our blog.
Sometimes, you need to run a program for a longer time than a single HTTP request. Use cases include:
- CPU-intensive jobs (e.g. video transcoding);
- running some code at a specified time, or on regular intervals;
- background activity (e.g. crawling 3rd party service to update you database);
- run a specific web server, like Node.js or Tornado;
- and much more!
DotCloud provides “worker services” dedicated to those tasks. There is a different service for each language: Ruby worker, PHP & PHP Worker, Perl Worker, Python worker, Node.Js... The only difference between them is the set of pre-installed packages, and dependencies handling: python-worker supports requirements.txt, while —for instance— ruby-worker supports Gemfile.
All worker services rely on Supervisor to start and monitor your processes. Supervisor will start defined programs automatically, and will restart them automatically if they crash or exit. If you just need to run a program at a specified interval, you can also use a crontab and ignore Supervisor.
You can also use a “non-worker” service to run some background jobs. More specifically, all services feature crontab, allowing you to run Periodic Tasks. So if you want to run a daily Python script just need to update, e.g., a stock ticker in your database, you don’t have to dedicate a python-worker to this task: you can setup the crontab in the same python service that you use for your web application. However, you should be aware that when you scale your application, the cron tasks will be scheduled in all scaled instances – which is probably not what you need! So in many cases, it will still be better to use a separate service.
Similarly, a lot of (non-worker) services already run Supervisor, so you can run additional background jobs in those services. Then again, remember that those background jobs will run in multiple instances if you scale your application. Moreover, if you add background jobs to your web service, it will get less resources to serve pages, and your performance will take a significant hit.
Technically, if you really want to know – there is almost no difference between worker and non-worker services. For instance, the python-worker service is basically the python service without Nginx (HTTP server) and uWSGI (Python web workers) running. Both can optionally run background processes using Supervisor.
Let’s assume that you are building a DotCloud application called “ramen”. For the sake of simplicity, we will also assume that everything related to this application is located in a directory called “ramen-on-dotcloud”.
Let’s setup our environment:
$ dotcloud create ramen Created application "ramen" using the flavor "sandbox" (Use for development, free and unlimited apps. DO NOT use for production.) $ mkdir ramen-on-dotcloud
A DotCloud application is described by a Build Files, which is a simple YAML file named “dotcloud.yml” located in our “ramen-on-dotcloud” directory. To add a new service to our app, we just add the following lines to “ramen-on-dotcloud/dotcloud.yml”:
worker: type: python-worker
With your DotCloud build file in hands you are ready to define your daemons.
Specify Python dependencies¶
Just like for the Python service, you can specify external dependencies in a file called requirements.txt. For example:
redis restkit == 2.3.3
If you specify a Python package installing executable scripts or binaries, they will be correctly installed in the $PATH and you will be able to run them without specifying their full path.
The python-worker service supports four different branches of Python (2.6, 2.7, 3.1, 3.2), it will default to Python 2.6 unless you specify otherwise. The current versions of each Python branch are listed in the table below. Pick the branch that works best for you.
* Python 2.6 is the default
To configure your service to use a version of Python other than 2.6, you will set the python_version config option in your dotcloud.yml.
For example, this is what you would have if you wanted Python 2.7.x:
www: type: python-worker config: python_version: v2.7