We removed our free Sandbox April 25th.
You can read more on our blog.

PHP-Resque

As you write your application you will certainly need to execute some asynchronous tasks. It could be anything that requires some form of (lengthy) processing: image resizing, archiving, document analysis...

These jobs could be run from the same machine were your application server is, but best practices advise to do this on a different machine because:

  1. You avoid impacts from the background jobs to your application;
  2. Decoupling parts of the application eases maintenance and scaling.

DotCloud makes it very easy to architecture your application on different “Services” each one with a different role and on a different machine. Background jobs match perfectly this feature: we will be using a Redis service to queue background jobs launched on one PHP service and executed on a second PHP service:

                   ╭─────╮        ╔═══════╗      ╭─────╮
                   │     │        ║       ║      │     │
                   │ PHP ╞═══════⇒  Redis ╠══════⇒ PHP │
                   │     │        ║       ║      │     │
                   ╰─────╯        ╚═══════╝      ╰─────╯
                   Enqueue                       Perform

The first PHP instance creates and enqueues jobs on Redis, which are dequeued and performed on a second PHP instances. This architecture is very scalable: you can have as much as PHP instances you want to perform or enqueue jobs and Redis could be replicated in a master/slave setup.

All of this is done with the help of a PHP library called PHP-Resque, which implements the job queues on Redis and offers an easy way to create and perform jobs. PHP-Resque is actually the PHP port of the Resque Ruby library used at GitHub for all their asynchronous jobs since 2009.

We are going to see how to use PHP-Resque through a simple web application. As a matter of facts the application also uses CodeIgniter a very straightforward web framework.

Application Architecture

You can clone the application from https://bitbucket.org/lopter/ci-resque/. Here is what the application directory looks like:

.
├── application/ # Hold our application code
├── dotcloud.yml # The description of our stack
├── index.php    # Entry point of the framework
├── nginx.conf   # Some Nginx rules to run the framework
├── php.ini      # Optional PHP configuration for development
├── postinstall* # Run at the end of the DotCloud build to setup PHP-Resque
├── static/      # Hold some CSS files
└── system/      # The framework “private” code

The relevant PHP code for PHP-Resque are all located in the “application” directory, we have:

  • a class to load PHP-Resque and connect it to the Redis Service;
  • a class to easily launch workers;
  • a class to display the content of the PHP-Resque queues as a web page;
  • a class to enqueue jobs from a web page.

We will see how to connect PHP-Resque to Redis and launch some workers first, then how to create jobs. There is also some DotCloud specific files that we are going to cover last.

Connecting PHP-Resque To Redis

PHP-Resque comes as a class called “Resque” but only with static methods to connect to Redis and enqueue jobs. We can wrap it up in a class called “Workers” to seamlessly add some functionalities:

  • Redis authentication support;
  • Redis credentials parsing from DotCloud’s environment.json;
  • A method to see the list of pending jobs on a queue;
  • A method to run the workers loop where PHP-Resque picks up and perform jobs.

The Redis connection and authentication is conveniently done from the constructor of the Workers class:

First, the constructor parses the Redis credentials from ~/environment.json:

<?php /* application/models/workers.php: */

    $this->_redis_host = self::_redis_credentials();

?>

This calls a “parsing” function which returns an associative Array with the Redis host, port and password. It is just a sequence of file_get_contents, json_decode and array_combine calls with some error handling:

<?php /* application/models/workers.php: */

    private static function _redis_credentials()
    {
        $json = file_get_contents('/home/dotcloud/environment.json');
        if ($json === false) {
            throw new WorkersException("Can't open environment.json");
        }

        $env = json_decode($json, true);
        if ($env === NULL) {
            throw new WorkersException("Can't decode the environment.json");
        }

        if (!array_key_exists('DOTCLOUD_SHM_REDIS_URL', $env)) {
            throw new WorkersException("Can't find redis in the environment.json, try to push again.");
        }

        return array_combine(
            array('scheme', 'user', 'pass', 'host', 'port'),
            preg_split('/[\/:@]+/', $env['DOTCLOUD_SHM_REDIS_URL'])
        );
    }

?>

Then, the constructor forwards those credentials to the Resque class and authenticates against Redis:

<?php /* application/models/workers.php: */

    Resque::setBackend("{$this->_redis_host['host']}:{$this->_redis_host['port']}", $this->_db);
    Resque::redis()->auth($this->_redis_host['pass']);

?>

Here is the complete constructor, $db is an optional parameter to use a particular Redis database number:

<?php /* application/models/workers.php: */

    public function __construct($db = 0)
    {
        parent::__construct();

        $this->_db = $db;
        $this->_redis_host = self::_redis_credentials();

        Resque::setBackend("{$this->_redis_host['host']}:{$this->_redis_host['port']}", $this->_db);
        Resque::redis()->auth($this->_redis_host['pass']);
    }

?>

Running The Workers

PHP-Resque provides a PHP script to run workers that will perform the jobs queued on Redis. Once again, we wrap that in our “Workers” class:

<?php /* application/models/workers.php: */

    public function run_workers()
    {
        require(APPPATH.'/third_party/resque/resque.php');
    }

?>

The workers are permanently running and polling the Redis queues. The workers are launched from another class that can be called through the CodeIgniter framework:

<?php /* application/controllers/jobs.php: */

    class Jobs extends CI_Controller {

        /**
         * Instanciate our Workers class.
         */
        public function __construct()
        {
            parent::__construct();

            $this->load->model('workers');
        }

        /**
         * Proxy method to run our workers.
         */
        public function run_workers()
        {
            $this->workers->run_workers();
        }
    }

?>

With this class in our hands we can run this command to launch our workers run from the root of the application:

$ QUEUES=* php index.php jobs run_workers

We will see how to this automatically when the code is pushed to DotCloud, but let’s create some jobs to perform first.

Create Some Jobs

Jobs in the PHP-Resque sense are just classes that define a “perform” method:

<?php

    class My_Job
    {
        public function perform()
        {
            /*
             * My_Job will be instantiated by a worker and its perform method
             * called.
             */
        }
    }

?>

The job class can also provide two optional methods “setUp” and “tearDown” that will be called before and after the “perform” method.

In our sample application jobs are created from the “Editor” class. It receives a filled HTML form and then use the enqueue method on our workers object to create a new job:

<?php /* application/controllers/editor.php: */

    $this->workers->enqueue(
            $this->input->post('queue_name'),
            'FakeJob',
            array('name' => $this->input->post('job_name'))
    );

?>

The enqueue method, in its turn, calls the Resque::enqueue static method that takes three parameters:

  1. The queue name that is extracted from the $_POST variable using one of the CodeIgniter function;
  2. The class name that should be used to actually instantiate a new job object on the worker process, here the class is “FakeJob”;
  3. An associative Array of parameters that will be passed back to the Job when it is performed, here our job will receive a “name” parameter.

DotCloud Specific Details

We have already seen how the “Workers” class uses the environment.json file to automatically connect to Redis. It is not the only DotCloud specific detail, there is also:

  • A DotCloud build file: dotcloud.yml;
  • A Nginx configuration include: nginx.conf;
  • A Supervisor configuration include: supervisord.conf that is installed from the postinstall hook.

The DotCloud Build File

The DotCloud build file is straightforward and just describes the architecture of our application on DotCloud, with its three services (a web server, a service to launch workers and a Redis service):

ui:
    type: php

workers:
    type: php-worker
    requirements:
        - proctitle

shm:
    type: redis

Notice the “proctitle” requirement on the “workers” service. This is an optional dependency of PHP-Resque to rewrite unix processes names so you can quickly see what the workers are doing when you run something equivalent to “ps aux”. This requirement correspond to a PECL package that will be installed automatically when you push the code on DotCloud.

The Nginx Configuration Include

We need to adjust how Nginx handle the HTTP requests and forward them to PHP, you can think of this file as a global .htaccess:

location ~ /system/.* {
    deny all;
}

location ~ /application/.* {
    deny all;
}

try_files $uri $uri/ /index.php?$args;

These three statements:

  • Deny access to the CodeIgniter framework and the application code;
  • Correctly rewrite the HTTP requests to CodeIgniter with “try_files” (this is actually needed for any PHP MVC framework).

The Supervisor Configuration Include

The Supervisor configuration include is used to launch and monitor a pair of PHP-Resque workers:

[program:experiment]
directory = /home/dotcloud/current/
command = php index.php jobs run_workers
environment = QUEUE=*
stopsignal = QUIT
stderr_logfile = /var/log/supervisor/%(program_name)s_error-%(process_num)s.log
stdout_logfile = /var/log/supervisor/%(program_name)s-%(process_num)s.log
numprocs = 2
process_name = "%(program_name)s-%(process_num)s"

You may have recognized a piece of .ini file, let’s break it down:

[program:experiment]

Defines a new background process configuration block. The process is called “experiment” here.

directory = /home/dotcloud/current/
command = php index.php jobs run_workers
environment = QUEUE=*

These three lines tells Supervisor how the background process should be launched:

  • In the directory “/home/dotcloud/current/” (this is where the DotCloud builder will install your application);
  • Using the command: “php index.php jobs run_workers”;
  • With the environment variable QUEUE set to * (you can use multiple queues with PHP-Resque and only launch workers on a specific queue), with * workers will process jobs on any queue.
stopsignal = QUIT

This is a hint to tell Supervisor to use the SIGQUIT Unix signal when it tries to gracefully stops the workers (instead of using the default SIGTERM convention). Since the workers will be stopped and started each time you push your code on DotCloud it is interesting to be able to gracefully interrupt the current jobs.

stderr_logfile = /var/log/supervisor/%(program_name)s_error-%(process_num)s.log
stdout_logfile = /var/log/supervisor/%(program_name)s-%(process_num)s.log

Redirect the output of the workers into these two log files, “%(program_name)s” will be replaced by “experiment” and “%(process_num)s” by the number of the worker.

numprocs = 2
process_name = "%(program_name)s-%(process_num)s"

Finally, this tells Supervisor to launch two concurrent workers, so we will be able to process two jobs at the same time.

Conclusion

PHP-Resque is very easy to get working on DotCloud. Let’s review what we did here:

  1. Connect PHP-Resque to Redis by using environment.json and Resque::setBackend("host:port");
  2. Authenticate to Redis by using: Resque::redis()->auth("password");
  3. Launched some workers by using the PHP-Resque worker script resque.php;
  4. Created a job class with a perform() method;
  5. Enqueued some jobs with: Resque::enqueue(QueueName, ClassName, Args).

If you want to dig into the short PHP files used here, you can clone the application and start to look into the application/controllers and application/models directories where the interesting parts are.

The example application lives on: http://php-resque.dotcloudapp.com/.

You can clone the code from https://bitbucket.org/lopter/ci-resque/, “cd” into it, create a your application on the flavor of your choice, and push it on your own DotCloud account with:

$ dotcloud create phpresque
Created application "phpresque" using the flavor "sandbox" (Use for development, free and unlimited apps. DO NOT use for production.)
$ dotcloud push phpresque

While you are fiddling with the web interface you can see the jobs being performed in the logs:

$ dotcloud logs phpresque.workers