T3X Generator for Subversion repositories

T3X is a proprietary file format used to distribute TYPO3 extensions.

This article describes how to create a T3X generator for your own Subversion (SVN) server, allowing you to dynamically create T3X packages out of the HEAD of any of your extension stored in a SVN repository.

Requirements:

  • Apache2 (or any other web server, you will have to adapt instructions accordingly)
  • mod_dav to serve your Subversion repositories
  • PHP5

We assert following:

This means we have a configuration similar to this one:

<VirtualHost *:80>
    DocumentRoot /var/www
 
    <Location /subversion/>
        DAV svn
        SVNParentPath /var/subversion/
        SVNPathAuthz off
        SVNAutoversioning off
        SVNIndexXSLT "/svnindex.xsl"
 
        # …
    </Location>
</virtualHost>

Let’s create a directory to hold our T3X generator and a default controller:

$ mkdir /var/www/t3x
$ cat <<EOT > /var/www/t3x/index.php 
> <?php
> echo 'Repository to use: ' . \$_SERVER['REQUEST_URI'];
> ?>
> EOT

Add a rewrite rule to redirect to our controller for every http://svn.yourdomain.tld/t3x/* URL: 

<VirtualHost *:80>
    DocumentRoot /var/www
 
    RewriteEngine on
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-l
    RewriteRule ^/t3x/ /t3x/index.php
 
    <Location /subversion/>
        # …
    </Location>
</VirtualHost>

Now restart Apache and make sure all invented URL starting with /t3x/ are properly redirected to this PHP controller.

Now that it works, here is the actual script used to dynamically create T3X packages from your SVN repository:

<?php
 
// Configuration [begin]
define('SVN_ROOT', '/var/subversion/');
define('SCRIPT_PATH', '/t3x/');
define('RM', '/bin/rm');
define('SVN', '/usr/bin/svn');
// Configuration [end]
 
require_once('class.typo3_lib.php');
 
class t3x_generator {
 
    protected $tempDirectory;
 
    /**
     * Default constructor
     */
    public function __construct() {
        $this->tempDirectory = sys_get_temp_dir() . '/' . md5(time() . rand());
    }
 
    /**
     * Clean-up routine.
     */
    public function __destruct() {
        if (is_dir($this->tempDirectory)) {
            exec(RM . ' -rf ' . $this->tempDirectory);
        }
    }
 
    /**
     * Processes a HTTP request and pushes the generated T3X to the
     * client or throws an exception in case of failure.
     *
     * @param string $request
     * @return void
     * @throws RuntimeException
     */
    public function process($request) {
        // Basic conformity check
        $this->checkRequest($request);
 
        $request = rtrim($request, '/');
        list($repository, $path) = explode('/', $request, 2);
 
        // We can only process TYPO3 extensions
        $this->checkTYPO3Extension($repository, $path);
 
        // Create the T3X package
        $this->checkout($repository, $path);
        $_EXTKEY = substr($path, strrpos($path, '/') + 1);
        $EM_CONF = array();
 
        require_once($this->tempDirectory . '/ext_emconf.php');
        $extInfo = array(
            'EM_CONF' => $EM_CONF[$_EXTKEY],
        );
 
        $uArr = typo3_lib::makeUploadArray($this->tempDirectory . '/', $_EXTKEY, $extInfo);
        $backupData = typo3_lib::makeUploadDataFromArray($uArr);
 
        $t3xFilename = $this->getT3xFilename($_EXTKEY, $path, $extInfo);
 
        // Send T3X to client
        header('Content-Type: application/octet-stream');
        header('Content-Disposition: attachment; filename=' . $t3xFilename);
        echo $backupData;
        exit;
    }
 
    /**
     * Ensures that $request is legitimate.
     *
     * @param string $request
     * @return void
     * @throws RuntimeException
     */
    protected function checkRequest($request) {
        if (!$request) {
            throw new RuntimeException('Empty repository', 1325667496);
        }
        if (!preg_match('@^[a-zA-Z0-9_-]+/([a-zA-Z0-9_-]+/?)+$@', $request)) {
            throw new RuntimeException('Unknown syntax', 1325667838);
        }
    }
 
    /**
     * Ensures that $repository is a valid SVN repository and that $path
     * points to a valid TYPO3 extension.
     *
     * @param string $repository
     * @param string $path
     * @return void
     * @throws RuntimeException
     */
    protected function checkTYPO3Extension($repository, $path) {
        if (!file_exists(SVN_ROOT . $repository . '/conf/svnserve.conf')) {
            throw new RuntimeException('Invalid SVN repository "' . $repository . '"', 1325668118);
        }
 
        $output = shell_exec(SVN . ' list file://' . SVN_ROOT . $repository . '/' . $path);
        $files = explode("\n", $output);
 
        if (!in_array('ext_emconf.php', $files)) {
            throw new RuntimeException('Invalid TYPO3 extension directory "' . $path . '"', 1325668221);
        }
    }
 
    /**
     * Checks-out a TYPO3 extension to the temporary directory.
     *
     * @param string $repository
     * @param string $path
     * @return void
     * @throws RuntimeException
     */
    protected function checkout($repository, $path) {
        $ret = -1;
        if (mkdir($this->tempDirectory, 0750, TRUE)) {
            exec(SVN . ' checkout file://' . SVN_ROOT . $repository . '/' . $path
                . ' ' . $this->tempDirectory, $output, $ret);
        }
 
        if ($ret !== 0) {
            throw new RuntimeException('Could not checkout the TYPO3 extension', 1325669951);
        }
    }
 
    /**
     * Returns a T3X package filename.
     *
     * @param string $extKey
     * @param string $path
     * @param array $extInfo
     * @return string
     */
    protected function getT3xFilename($extKey, $path, array $extInfo) {
        // TODO: create a meaningful filename according to $path (tag name, ...)
        return 'T3X_' . $extKey . '-' . str_replace('.', '_', $extInfo['EM_CONF']['version'])
            . '-z-' . date('YmdHi') . '.t3x';
    }
}
 
 
$t3x_generator = new t3x_generator();
try {
    $svnUrl = trim(substr($_SERVER['REQUEST_URI'], strlen(SCRIPT_PATH)));
    $t3x_generator->process($svnUrl);
} catch (RuntimeException $e) {
    header('HTTP/1.0 400 Bad Request');
    echo $e->getMessage();
    die();
}
 
?>

This script requires a library with a few functions from TYPO3 sources which are not shown here. Download   t3x-generator.tar.gz (4 KB) and unpack it to /var/www/t3x/.

Last but not least: You certainly should consider securing access to this /t3x/ with same restrictions as for /subversion/ (typically with your company’s LDAP), if needed of course :)

Mirror your extension from Forge

Forge is really useful to host your extension and allow virtually anyone to contribute to your project but it currently lacks a way to generate T3X packages as described in this article. If you manage your own Subversion server, you can easily mirror any extension from Forge (or any other server actually) to your own Subversion server and generate T3X packages from there…

Let’s say you want to synchronize  extension egovapi. We will use the command svnsync to mirror one of the extension from Forge.

svnsync requires a hook to be created in our local SVN repository to allow the user running svnsync to arbitrarily modify the repository revisions.

Note: svnsync does not allow to sync two or more projects from the master repository

Beware: we will allow any Subversion user to do that. For your real SVN repository, be sure to use an enhanced script as shown in the  SVN red book to at least restrict who is able to commit to your mirror repository.

OK, let’s create a local repository and a minimal pre-revprop-change hook for it:

$ svnadmin create /var/subversion/egovapi-mirror
$ cat <<EOT >/var/subversion/egovapi-mirror/hooks/pre-revprop-change
> #!/bin/sh
> exit 0
> EOT
$ chmod +x /var/subversion/egovapi-mirror/hooks/pre-revprop-change

That’s it, now we can initialize the mirroring to synchronize egovapi from Forge to our local “egovapi-mirror” SVN repository:

$ svnsync init file:///var/subversion/egovapi-mirror https://svn.typo3.org/TYPO3v4/Extensions/egovapi
$ svnsync sync file:///var/subversion/egovapi-mirror

OK, as some of our TYPO3 friends would say you should consider go grab a Caffè Latte or, if it’s afternoon or late night already, a few shots of Espresso as the initialization process will just last forever with Forge as all extensions are stored in the very same SVN repository and we have to sync all revisions and filter out what is not related to our extension egovapi…

When it’s done, well, here you are! Enjoy and svnsync sync again from time to time…

Of course, one better solution would be to SVN checkout directly from Forge without syncing locally at all. This is a minor adjustement to the T3X generator you should be able to do easily :D

 

PS: If sync aborts for whatever reason, you may have a problem to restart it:

Failed to get lock on destination repos, currently held by ’svn01:7d14e163-a3f3-4a29-a829-25ca6bdf3296’

If so, just issue this command and sync again to continue where it stopped:

$ svn propdel -r 0 --revprop svn:sync-lock file:///var/subversion/egovapi-mirror/
Flattr