This little snippet will also allow you to capture your query string variables should the need arise.

url.rewrite-once = (    
    "^/(css|files|img|js)/(.*)" => "/$1/$2",
    "^/([^?]*)(?:\?(.+))?$" => "/index.php?url=$1&$2"
)

Posted via email from shakeit

Disclaimer: I am not a database architect nor a certified database programmer. I might not be database guru, but I have been working with SQL Server long enough to know what works and what doesn’t.

I’m not a database programmer per say, but I’m expected to be one at work (even though I’m a code jockey). I don’t mind it most of the time, but there are times when it becomes really frustrating. Especially when you come across some schema that is poorly implemented. Normalization is a double edged sword. On one side, normalization is good for data integrity. On the other, it becomes a pain in the arse to get at your data. There are ways around the later (Views, de-normalized tables), but, a pain in the arse it still is. The one thing that really bugs me is normalization done wrong. We use Microsoft SQL server at work, but this should apply to any relational database.

Recently, at my day job, I’ve been working on a particular project which involved consuming data from a third party for use in a .NET web service. We were already consuming data from this vendor but they recently changed the schema (and data) and are deprecating the old format. The new feed is basically a complete dump of their data and schema. Not all of their data, just data we subscribe to (about 1M records total). Instead of modifying our import process to accommodate the current (old) schema (which would have been a huge effort on our part), we just created a new import process and modified the web service.

After working with this new schema for a couple weeks, I have come to the conclusion that, well, it’s crap. First and foremost, it’s not normalized properly. If you have an column in a table that looks like it should be a foreign key, then by all means, make it so. For example, say I have a table called Widgets with several columns. One of those columns is called WidgetTypeId and another called WidgetTypeDisplayName. There’s a prime candidate for another table. I don’t care if you only have two WidgetTypes, it belongs in its own table. Here’s why: When selecting records out of the table and filtering on WidgetTypeDisplayName, it gets really expensive. Sure, you could add an index on that column, but it still would be slower compared to joining on a WidgetType table. If I had ten million widgets and only 100 widget types, I’m only filtering 100 records vs. ten million.

What about filtering on the WidgetTypeId column? BZZZZZ. Where did you get that ID? Did you select it from the same table? Guess what, same difference. Maybe you hard-coded it in the query. That’s not smart either, especially if you have multiple environments that could possibly be out of sync (i.e. WidgetTypeId 4 in development might not be associated to the same thing in production). Put it in it’s own table.

I ran into this several times with this new schema at work and it was a complete pain. For one thing, in development, the queries were always relatively quick to return. In production, however, not the same. This database sat on the same server as another production database so there was more load on it. Enough to make the queries perform poorly. On the order of 10 to 40 seconds to return data in some cases. That’s not good. My solution: create a single de-normalized table (for some reason, we don’t use views) of commonly used data points. The result was millisecond returns and simple queries. Furthermore, instead of filtering/joining on tables with half a million records, it now filters on less than 20 thousand, properly indexed records.

I’ve been using Amazon’s S3 service for a couple months now. It was working OK using s3sync and a cron job, but it seemed like it wasn’t actually making incremental backups and I wasn’t 100% sure that it was backing up everything (i.e. it appeared to be crapping out once in a while). I searched around for various S3 backup solutions and found a handy utility called duplicity. Even more handy that it is available for most distributions (Archlinux, the debs, and Fedora anyway).

From the duplicity home page:

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

What you’ll need

You’ll need to make sure you have a few things installed before you install duplicity. Namely librsync and GnuPG. Luckily, if the duplicity package is available for your distribution, you probably needn’t worry.

Here’s a rundown of the steps involved:

1. Generate a new GnuPG key
2. Create a simple shell script wrapper
3. Create a cron job

Generating a new Key

Start by generating a new gpg key for duplicity. Or if you have an existing one, you can use that.

N.B. I set this up on a Slice running Arch64 and had problems generating a new key (gpg --gen-key). Apparently, it could not generate enough entropy. Not a problem though: Just generate the keys else where and import them later if this happens to you.

#~ gpg --gen-key
gpg (GnuPG) 1.4.7; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.


Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection?

Default (DSA and Elgamal) is fine here.

DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)

The default (2048) is more than enough for this. Change it to whatever you want.

Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)

Unless you want the key to expire (I don’t see why one would want that), the default is what we want.

Key does not expire at all
Is this correct? (y/N)

Um, yes, this is correct.

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"


Real name: DuplicityBackup
Email address: duplicity@mydomain.com
Comment: Key for Duplicity
You selected this USER-ID:
    "DuplicityBackup (Key for Duplicity) <duplicity@mydomain.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?

Enter whatever information you want here and type ‘O’ for ‘Okay’

You need a Passphrase to protect your secret key.


Enter Passphrase:

Enter something. Anything. The more complex the better. This is your private data. Remember that it’s being transfered over http to a server you don’t own. I don’t care if it is Amazon. Remember what you type because you’ll need it later while creating the wrapper script.

gpg: key 9929DAB1 marked as ultimately trusted
public and secret key created and signed.


gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
pub   1024D/9929DAB1 2007-11-15
      Key fingerprint = 3378 8E93 4349 0E7F 44F3  7C81 2460 5A11 9929 DAB1
uid                  DuplicityBackup (Key for Duplicity) <duplicity@mydomain.com>
sub   2048g/5385A6BB 2007-11-15

And you’re done. Make note of the key (in this case, 9929DAB1) as we’ll need that later too.

But I already have a key I want to use

OK, fine, but chances are, if you have a key already, you know how to get it. However, if you don’t know how to get your key, gpg --list-keys. You want the key in the ‘pub’ line… after the forward slash ‘/’

The Wrapper

This can be written in any language really. I chose shell because it’s easy and basic. You could run the duplicity now on the command line, but writing a wrapper is much more convenient and makes adding a cron job later a lot easier. Here’s what you’ll need:

• Your Amazon S3 Access Key ID and Secret Access Key. If you don’t have one, you’ll have to sign up for one.
• Your GPG key
• Your GPG key’s passphrase
• A list of directories you want to back up

Here’s a basic script that works for me:

#!/bin/bash
# Export some ENV variables so you don't have to type anything
export AWS_ACCESS_KEY_ID=<your-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export PASSPHRASE=<your-gpg-passphrase>

GPG_KEY=<your-gpg-key>

# The source of your backup
SOURCE=/

# The destination
# Note that the bucket need not exist
# but does need to be unique amongst all
# Amazon S3 users. So, choose wisely.
DEST=s3+http://<your-bucket-name>

duplicity
    --encrypt-key=${GPG_KEY} \
    --sign-key=${GPG_KEY} \
    --include=/boot \
    --include=/etc \
    --include=/home \
    --include=/root \
    --include=/var/lib/mysql \
    --exclude=/** \
    ${SOURCE} ${DEST}

# Reset the ENV variables. Don't need them sitting around
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=

And, that’s pretty much it. Save the file as something creative, like, backup and make it executable (chmod 700 backup). If you want to test it first (and you have the disk space), change the destination to some /tmp directory or external HDD. Once you’ve got it working the way you want, set it up as a cron job. Daily, weekly, monthly… doesn’t matter.

Duplicity is a nice backup solution for any situation, not just Amazon’s S3. It can handle HTTP, SCP and local backups as well. I highly recommend reading the duplicity man page and checking out the various command line arguments and availble options.

A couple of Thanks goes out to Tim McCormack’s and Ben and Ron’s articles which got me started.

Tim points out that, adding your GPG PASSPHRASE to the shell script might not be the most secure method, especially in a shared environment. I agree, however, it kind of defeats the purpose of automated backups if you have to actually enter your passphrase (twice) on the command line when calling the wrapper script. One way I managed to go around this is to create a simple C++ application that prints the passphrase.

Here’s the C++ code:

#include <stdio.h>
int main()
{
    printf("your-gpg-passphrase");
    return 0;
}

Compile

#~ gcc gpg-passphrase.c -o gpg-passphrase

Make it executable by your user and set the sticky bit so no one else can execute it

#~ chmod 700 gpg-passphrase


~ chmod +s gpg-passphrase

Modify the wrapper script to use the binary for the passphrase

export PASSPHRASE=$(gpg-passphrase)

You might go as far as to do the same thing for your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as well. There are probably other ways around this, but this was a quick a dirty way to not have readable strings in shell scripts. I figure, if someone has rooted my server, I’ve got bigger problems to worry about than my data sitting on Amazon’s S3.

I recently acquired a 256 Slice from Slicehost. While their distribution selection is good, I was hoping for something, well, different. My old VPS is rocking Ubuntu (quite well) but I just want something else. Slicehost’s other distributions weren’t going to cut it.

1. Ubuntu Dapper – Already running that.. that’s how I got here.
2. CentOS 4.3 – Um, no. Never liked RedHat. Hate RPMs. Package management is a nightmare.
3. Gentoo 2006.1 – I’ve used Gentoo in the past (a lot) and enjoyed it. I even tried it on my old VPS. I just don’t have the patience to wait for shit to compile.
4. Debian Etch – Debian is a true soldier. Secure and stable. However, not up-to-date enough for my tastes (that’s why I went with Ubuntu).
5. Fedora 6 – See #2.

Enter Archlinux.

This little how-to was taken from various places on the net, but the bulk of it is from Anders’ Blargh post and a couple of comments in a previous post of mine. So, let’s dive in.

Back It Up!

If you have any valuable information on your slice, now would be the time to make a backup of the important stuff you want to keep.

Bootstrapping

First thing you want to do is bootstrap Arch64 to directory on your current distribution. If you happen to have a 64bit version running at home, you could do this locally and it should still work (in theory).

# export PACMAN_VERSION=3.0.5-2


export XEN_MODULES_VERSION=2.6.16.29

wget http://www.randys.org/assets/2007/8/17/archbootstrap-packages-20070817

wget ftp://ftp.archlinux.org/current/os/x86_64/pacman-${PACMAN_VERSION}.pkg.tar.gz

tar zxvf pacman-${PACMAN_VERSION}.pkg.tar.gz -C /

mkdir /arch

At this point, you might want to edit /etc/pacman.conf and /etc/pacman.d/current to your liking. Picking a close/fast mirror will help speed things along.

# pacman.static -Sy cat archbootstrap-packages-20070817 -r /arch

Now you need to bind /dev, /sys and /proc to the bootstrap directory so you can chroot in.

# for s in dev sys proc; do mount /$s /arch/$s --bind; done

Copy some important files for the chroot.

# cp /etc/mtab /arch/etc/


cp /etc/fstab /arch/etc/

cp /etc/resolv.conf /arch/etc/

We also need to copy the all important xen kernel modules from the existing slice into our Arch64 bootstrap directory.

# cp -r /lib/modules/${XEN_MODULES_VERSION}-xen /arch/lib/modules/

Now we’re ready to chroot into your new Arch64 directory.

# chroot /arch /bin/bash

One of gotchas I had was when I went to SSH into my newly booted Arch64 Slice. It would connect and I could enter my password, but it would hang and never drop me to a command line prompt. Did a little searching and found that it was all because /dev/pts was not being mounted at boot time. Easy fix.

#[] echo "none  /dev/pts    devpts  gid=5,mode=620  0 0" >> /etc/fstab

Edit /etc/locale.gen and enable any locales you require. en_US.UTF-8 UTF-8 and en_US ISO-8859-1 are popular choices. Now generate the locales.

#[] locale-gen

Edit /etc/pacman.conf and /etc/pacman.d/(current|extra|community) in your chroot to your liking. If you don’t do it here, you can always do it when you get it running. But we’re about to install openssh, so now is as good a time as any.

#[] pacman -Sy openssh

Now you nee to add sshd to our DAEMONS array in /etc/rc.conf file

#[] vim /etc/rc.conf
    ...
    DAEMONS=(syslog-ng network netfs crond sshd)
    ...

While you’re in /ect/rc.conf, go ahead and setup your network interfaces and gateway. You can get the current information from your running Slice (for Ubuntu cat /etc/network/interfaces)

    ...
    eth0="eth0 XXX.XXX.XXX.XXX netmask 255.255.255.0 broadcast XXX.XXX.XXX.XXX"
    INTERFACES=(lo eth0)
    gateway="default gw XXX.XXX.XXX.XXX"
    ROUTES=(gateway)
    ...

Next, you’ll need to modfy /etc/hosts.allow so you can actually connect to our new Arch64 VPS via ssh.

#[] echo "sshd : ALL : allow" >> /etc/hosts.allow

If there’s any other tweaks you want to make, make them now. Otherwise, exit the chroot.

One thing I did before exiting the chroot to cut down on the bootstrap size is to clear pacman’s cache directory. It should probably save you 100MB+ and there’s no need to keep them at this point.

#[] pacman -Scc


[] exit

Unmount the dev, sys and proc filesystems you mounted earlier.

# umount /arch/dev


umount /arch/sys

umount /arch/proc

Now we’ll archive the files.

# cd /arch && tar cjvf /archbootstrap-$(date +%Y%m%d).tar.bz2 *

That’s it. You’re all set. If you want to keep a copy of this so you don’t have to do all this again, do it now.

# scp /archbootstrap-$(date +%Y%m%d).tar.bz2 <username>@<hostname>:<path>

Dealing with Slicehost

At this point you should have a working Arch64 install ready to be installed. Now you need to login to the Slicehost manager and put your slice into Rescue Mode. Login to your slice via SSH. Once logged in you’ll need to mount your slice’s partition and copy the archive to the rescue disk.

#[r] mkdir /mnt/oldslice


[r] mount /dev/sda1 /mnt/oldslice

[r] cp /mnt/oldslice/archbootstrap-* /root

[r] umount /mnt/oldslice

Next, format your old slice partition and re-mount it so you can install Arch64.

#[r] mke2fs -j /dev/sda1


[r] mount /dev/sda1 /mnt/oldslice

[r] tar xjvfp /root/archbootstrap-*.tar.bz2 -C /mnt/oldslice/

And that’s about it. You should now have a working Arch64 installation on your new Slice! Probably a good idea to double check all your settings before rebooting your slice, but, then you do have access to the rescue console.

Enjoy! And thanks to Anders for pointing me in the right direction.

If you find any errors or have any additions, please let me know and I’ll update this post.

One of the odd things I’m finding with Mephisto is that none of the templates I looked at had monthly archives in the sidebar. That’s generally not that strange, but Mephisto doesn’t really paginate (out of the box, anyway) sections (categories), so the lack of any way for a user (that would be you) to navigate to older posts is beyond me. So the search for some Liquid code to display monthly archives began.

Mephisto’s documentation is seriously lacking, but what can you expect from a small development team with no funding behind the app. Most of the links I followed were to the Mephisto Google Group. I took this snippet from one particular thread:

 <ul>
 {% for month in section.months %}
     {{ section | monthly_articles: month | assign_to: 'monthly_articles' }}
     <li>
     {% if monthly_articles %}
         {{ section | link_to_month: month }} ({{ monthly_articles | size }})
     {% else %}
         {{ month | format_date: 'my' }}
     {% endif %}
     </li>
 {% endfor %}
 </ul>

But this would still spit out months with no posts. We can’t have that! So, I turned it into this:

{% if section.months.size > 0 %}
<ul>
    <li class="head">Archives</li>
    {% for month in section.months %}
    {{ section | monthly_articles: month | assign_to: 'monthly_articles' }}
        {% if monthly_articles.size > 0 %}
            <li><span class="right">{{ monthly_articles | size }}</span>
            {{ section | link_to_month: month }} </li>
        {% endif %}
    {% endfor %}
</ul>
{% endif %}

Much better! Now I have this:

So, I’ve been reading up a bit on prototype.js and its Ajax helpers. It’s an amazing tool and helped me write the bit of info at the top of the page. It’s pretty basic, but here’s the code that does most of the work:

function lastfm()
{
    new Ajax.Request('/as/recenttracks.xml',
    {
        method: 'get',
        onLoading: function() {


    },
    onLoaded: function(transport) {
        if (transport.overrideMimeType) {
            transport.overrideMimeType('application/xml');
        }
    },
    onSuccess: function(transport) {
        var response = transport.responseXML.documentElement;
        updateLastfm(response);
    },
    onFailure: failedLastfm()
});


}

The only issue I ran into was that I was originally using the RSS flavor of recent tracks, however it didn’t split up the artist and track information. It displays it as <title>[artist] – [track]</title>. That en-dash in the middle was preventing me from using title.split() on the JavaScript side of things. Really weird.

Also, since I’m a complete newbie with Ruby, I couldn’t figure out how (read: didn’t take the time to learn) to grab the content from a remote server and serve it up to the JavaScript. I’m sure it’s pretty simple… but I was at work and in a hurry. So, being that I know PHP, I just created a script to download the file and save it to the local disk and setup a cronjob.

Here’s the PHP script:

class lastfm
{
    private $user;
    private $reports;
    private $basews;
    public $saveto;


function __construct($user)
{
    $this->user = $user;
    $this->basews = 'http://ws.audioscrobbler.com/1.0/user/';
    $this->reports = array(
        'recenttracks.xml',
        'weeklyartistchart.xml',
        'weeklytrackchart.xml',
        'topartists.xml'
    );
    $this->saveto = '.';
}

public function go()
{
    for ($i = 0; $i < count($this->reports); $i++)
    {
        try
        {
            $opts = array('http' => array('method' => 'GET', 'header' => 'Content-type: text/plain; charset=utf-8'));
            $context = stream_context_create($opts);

            $fp = fopen($this->saveto.DIRECTORY_SEPARATOR.$this->reports[$i], 'w+');
            $stream = fopen($this->basews.$this->user.'/'.$this->reports[$i], 'r', false, $context);
            $string = stream_get_contents($stream);
            fwrite($fp, $string);
            fclose($fp);
            fclose($stream);
            echo 'Saved ' . $this->saveto.DIRECTORY_SEPARATOR.$this->reports[$i] . "\n";
        }
        catch (Exception $e)
        {
            echo $e->getMessage() . "\n";
        }
    }
}

public function setSaveto($path)
{
    if (is_dir($path))
    {
        if (ereg('\/$', $path))
        {
            $path = ereg_replace('\/$', '', $path);
        }
        $this->saveto = $path;
    }
    else
    {
        $this->createDir($path);
        $this->setSaveto($path);
    }
}

private function createDir($path)
{
    if (!is_dir($path))
    {
        $res = `mkdir -p $path`;
    }
}


}

I’ll hit up the Rails API Docs one of these days and write a simple Ruby script that does all the work of the PHP script. Even better would be to process the XML document using ruby that just returns a string of HTML and use Ajax.PeriodicalUpdater($(e), ...).

When I switched from Wordpress to Typo, I faced the issue of keeping my Wordpress permalinks in Typo. Unfortunately, there’s no simple way of doing this. I noticed the ‘redirects’ table in the database, but I could never get it to realy do anything. So, I had to dig in and find the bit of code that controlled the permalinks in Typo.

Typo uses ‘articles’ as it’s base for all posts so I looked in config/routes.rb and found what I needed to change. As you can see from the url of this post, I use ‘content’ as the base of my permalink. So, I changed all the references of ‘articles’ to ‘content’ and all seems to be right in the world.

I changed

1
2

map.connect 'articles',
    :controller => 'articles', :action => 'index'

to

1
2

map.connect 'content',
    :controller => 'articles', :action => 'index'

and then changed

1
2
3

map.connect 'articles/:year/:month/:day',
    :controller => 'articles', :action => 'find_by_date',
    :year => /\d{4}/, :month => /\d{1,2}/, :day => /\d{1,2}/

to

1
2
3

map.connect 'content/:year/:month/:day',
    :controller => 'articles', :action => 'find_by_date',
    :year => /\d{4}/, :month => /\d{1,2}/, :day => /\d{1,2}/

and so on.

The only thing I haven’t gotten to work is to map the pages. I can change the ‘pages’ map, but it doesn’t act as desired. I only had 3 pages in Wordpress, so I might as well either alias them or premanent redirect in the Apache configuration.

If anybody has some insight, I’d love to here it. Also, still haven’t figured out why mod_rewrite is working like it should.

Wow. I’ve been testing the possibility of running Typo on my VPS with the pre-notion that it just wouldn’t work very well. I think I may be wrong. It’s been running for about a week while I tweak some shit and so far, my memory performance has been stable. Of course, thing would probably be different if I threw some real traffic at it… but, then again, I don’t get a lot of that.

The Setup

Initially, I was going to use lighty + fastcgi to serve up the Rails application. So, I had setup lighty to serve up my PHP/MySQl sites and that worked just fine. Then I started reading up on Mongrel and wanted to see what all the hubub was about. But then I got to reading the Mongrel site (more specifically the part where he says don’t use lighty and mod_proxy).

Crap.

Aight, so back to Apache and their proxy setup. no biggie. I racked my brain for several hours one night trying to figure out why the hell I kept getting these 403 Forbidden errors in Apache. Hours. After the millionth Google search, I finally found the issue: The mod_proxy configuration in Ubuntu is turned off by default (well, not really turned off… it just denies traffic to the proxy server).

 8     <Proxy *>
 9         Order deny,allow
10         #Deny from all
11         Allow from .randys.org
12     </Proxy>

Line 8 was initially not commented out. I had to comment that out and add the Allow from .randys.org bit.

Once I got that changed and reloaded Apache, everything is working nicely together.. even PHP.

The Tweaks

So far, the only tweaks I’ve managed to make is to the flickr Sidebar plugin. It now uses Lightbox V2. That was a bit tricky, considering I don’t really know Ruby but, it all worked out in the end.

As I figure out more stuff to do, I’ll post more.

In following with the LAMP, MAMP, and WAMP themes, I’ve come up with my own acronym: LiMP. Lighttpd, MySQL, PHP. Of course, this doesn’t really follow the conventions of the other acronyms (my OS isn’t represented). Mainly because adding an ‘M’ just doesn’t sound (or look) right. It’s a great setup and I recently reconfigured it so it’s a (somewhat) isolated installation that could potentially be installed on any OS X system. I also managed to get ExecWrap working properly as well.

*NB* This is a fairly technical article and requires getting your fingers dirty in the Terminal (a.k.a. comman line). If you’re not fully comfortable in the Terminal, I suggest you familiarize yourself with the Terminal. You’ll also need to have installed the latest version of Xcode.

You can build this just about anywhere on your system you like. I personally keep everything in

/usr/local/src

sudo -s

Create your src directories:

mkdir -p /usr/local/src && cd /usr/local/src

That’s it. Now lets dig in…

Setting up Lighttpd on OS X

I figured building this on OS X wouldn’t take too much effort and I was pretty much right. Lighttpd builds just fine on OS X but it does need some other libraries installed for certain functionality. Specifically, fastcgi and Perl Compatible Regular Expressions. These libraries install without issue as well.

First lets grab the fastcgi libraries:

curl -O http://www.fastcgi.com/dist/fcgi-2.4.0.tar.gz
tar zxvf fcgi-2.4.0.tar.gz && cd fcgi-2.4.0
./configure
make
make install
cd ..

Now lets get the PCRE libraries:

curl -O ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-5.0.tar.gz
tar zxvf pcre-5.0.tar.gz && cd pcre-5.0
./configure
make
make install
cd ..

Now for lighttpd:

curl -O http://www.lighttpd.net/download/lighttpd-1.4.12.tar.gz
tar zxvf lighttpd-1.4.12.tar.gz && cd lighttpd-1.4.12
./configure --prefix=/Library/limp/lighttpd

Hopefully, you’ll see something like this after the configure script is done.

Plugins:


mod_rewrite     : enabled
mod_redirect    : enabled
mod_ssi         : enabled
mod_cgi         : enabled
mod_fastcgi     : enabled
mod_proxy       : enabled
mod_evhost      : enabled
mod_simple_vhost: enabled
mod_mysql_vhost : enabled
mod_access      : enabled
mod_alias       : enabled
mod_setenv      : enabled
mod_usertrack   : enabled
mod_compress    : enabled
mod_auth        : enabled
mod_status      : enabled
mod_accesslog   : enabled
mod_rrdtool     : enabled
mod_secdownload : enabled
mod_expire      : enabled

If you don’t see mod_fastcgi in there, something went south.

make
make install
cp doc/lighttpd.conf /Library/limp/lighttpd/lighttpd.conf

MySQL

For MySQL, I just used the standard binary installation provided by MySQL. Therefore, we’ll need to also configure and install the libraries after installing the binary package. (Thanks to Richard Valk for his article series for this bit).

First, download and install MySQL (version 5.0.24a as of this writing) Standard binary for OS X for your platform (PPC or Intel). Once you’ve installed the package, install the StartupItem and make your life simple-er (??). After everything is setup, you should modify your

PATH

export PATH="/usr/local/bin:/usr/local/mysql/bin:"${PATH}

Now we have to rebuild mysql and install the shared libraries we need for building PHP. Make sure you’re still in your ’src’ directory.

curl -O http://www.stathy.com/mysql/Downloads/MySQL-5.0/mysql-5.0.24.tar.gz
tar zxvf mysql-5.0.24.tar.gz && cd mysql-5.0.24
./configure --prefix=/usr/local/mysql \
 --localstatedir=/usr/local/mysql/data \
 --libexecdir=/usr/local/mysql/bin \
 --libdir=/usr/local/mysql/lib \
 --with-server-suffix=-standard \
 --enable-thread-safe-client \
 --enable-local-infile \
 --enable-shared \
 --with-zlib-dir=bundled \
 --with-big-tables \
 --with-readline \
 --with-archive-storage-engine \
 --with-innodb \
 --without-docs \
 --without-bench \
make
make install
cd ..

Compiling PHP on OS X (with the mysqli extension)

This was fairly straight forward with the exception of GD. It took me a while to figure this out, but I was using

--with-gd=/sw

--with-gd

./configure \
 --prefix=/Lbrary/limp/php \
 --enable-fastcgi \
 --enable-force-cgi-redirect \
 --enable-mbstring \
 --with-xml \
 --with-zlib \
 --with-curl \
 --with-mysql=/usr/local/mysql \
 --with-pdo-mysql=/usr/local/mysql \
 --with-mysqli=/usr/local/mysql/bin/mysql_config \
 --with-pdo-sqlite \
 --with-sqlite \
 --with-mcrypt=/sw \
 --with-gd \
 --with-jpeg-dir=/sw \
 --with-png-dir=/sw \
 --with-zlib-dir=/sw \
 --with-xpm-dir=/usr \
 --enable-exif \
 --enable-ftp \
 --enable-libxml \
 --enable-soap \
 --enable-sockets


make && make install
cp /Library/limp/php/bin/php /Library/limp/php/bin/php-cgi

I’m renaming the php binary to php-cgi because, well, that’s what it is. It’s the CGI version, not the CLI version. If you want to compile the CLI version, replace

 --enable-fastcgi \
 --enable-force-cgi-redirect \

with

 --enable-cli \

and run

make && make install

Building ExecWrap

I’ll have to finish this at a later date… sorry. This was originally posted to my old Wordpress blog as a Draft but got imported into Typo via the wordpress2.rb script as a published article… so I left it.

Part of my decision to change hosting providers was to expand my knowledge of technologies. I know how to write PHP, SQL and a whole host of other languages. What I was less familiar with was the servers that run them and other parts of a hosting system (mail, dns et al). Switching to a VPS setup allowed me to explore my options in what I would run on my system and fine tune the processes to run under limited resources.

I was already familiar with Apapche and how to set that up with PHP. Apache2 makes it really easy to setup suExec with mod_suphp. Simply add “SuPHP_UserGroup $user $group” to a virtual host and viola, all PHP processes run as that user (as fastcgi). That was great and all, but on a system with limited resources, apache is dog. It sucks up way too much memory. After setting up my VPS and running all the services, I was up to about 150MB of RAM used. That’s with apache2, php, mysqld, postfix, postgrey, courier-imapd (and ssl), courier-pop3d (and ssl) and mailman (which is another memory hog, but that’s another post) running. Granted 150MB isn’t that bad for a web server, especially if you have an entire system to yourself that has 1 or 2 GB of RAM. I’m on a VPS with a mere 256MB of RAM.

Enter Lighty

Lighttpd is an open source, fast and efficient alternative to Apache. It pretty much does everything Apache does but with a much smaller footprint. Yes, it was a little more difficult to setup, but most of my troubles came from not knowing the Lighty configuration syntax. It’s not hard to master, just different than Apaches familiar tag based config files.

So far, this is what I have running on my VPS:

lighttpd (1.4.11)
php* (5.1.4) (+fastcgi)
mysql (4.1.21)
postfix (2.2.10)
postgrey (1.24)
courier (4.0.4)
openssh (4.3_p2)
tinydns (1.05)
proftpd (1.2.10)

Current memory usage:

Total: 239 MB   Used: 96 MB   Free: 143 MB

UPDATE: I recently added a Typo blog (RoR application) to one of the domains I’m hosting and my memory usage jumped a little… well, a lot really. I’m probably sitting at about 120-130 MB used at the moment.

And now the HOWTO

Configuring lighty wasn’t that hard. The hardest part was figuring out things like setting up rewrite rules for web applications like Wordpress and Gallery2 search engine friendly URLs. The other tricky part was getting ExecWrap (similar to Apache’s suExec wrapper) working properly. Well, it wasn’t that tricky, I just had some settings wrong so it appeard to be tricky. Let’s tackle the ExecWrap part first.

ExecWrap Your PHP

You’ll need to grab, build and install the ExecWrap wrapper first. It’s actually pretty straight forward. The important part is setting the correct permissions on the files involved in this setup (and using the correct UIDs and GIDs for the wrapper). For the sake of this post, I’ll skip that part. If I get enough questions about it, I’ll post a follow up on how to set this up properly.

So, here’s my setting for PHP/FastCGI setup on lighty:

fastcgi.server = (
    ".php"  => ((
            # socket - this needs to be writable by the webserver itself
            "socket"            => "/var/run/fastcgi/fastphp.socket",
            # bin-path - the path to the execwrap script -- see NB below
            "bin-path"          => "/usr/lib/php5/bin/execwrap",
            # check-local - Not 100%, but I'm pretty sure this
            # disables cheking that the local file exists
            "check-local"       => "disable",
            # max-procs - Maximum number of procs to fire up.
            # I'm pretty stingey here, but my site doesn't see
            # a lot of traffic.
            "max-procs"         => 1,
            # bin-environment
            "bin-environment"   => (
                # Howman PHP_CFGI_CHILDREN to start up
                "PHP_FCGI_CHILDREN"     => "4",
                # Maximum request (per child? i dunno)
                "PHP_FCGI_MAX_REQUESTS" => "1000",
                # UID - User ID you want the script to execute as
                "UID"                   => "1000",
                # GID - Group ID you the script to execute as
                "GID"                   => "1000",
                # TARGET - the actual script to run
                "TARGET"                =>  "/usr/lib/php5/bin/randy.php.sh",
                # CHECK_GID - this just checks the GID of the wrapper script
                "CHECK_GID"             => "1"
            ),
            # Copied from another site... not quite sure what it
            # does other than copying those env to $_ENV
            "bin-copy-environment"  => ("PATH", "SHELL", "USER"),
            # Fixes broken $_SERVER['PATH_INFO] I believe
            "broken-scriptfilename" => "enable"
        )
    )
)

The contents of randy.php.sh:

#!/bin/sh
exec /usr/lib/php5/bin/php-cgi

NB: Note that the execwrap script must be executable by lighty and must also have the SUID bit set. Also, the shell script needs to be owned by the user in which you wish to execute PHP as (in my case, my username). Also note that execwrap can live anywhere you specify when you compiled the script. In my case, I specified in the execwrap_config.h /usr/lib/php5/bin as the path where it will live. The shell script must also live under the same path.

Wordpress & Gallery URLs

Permalinks. The best thing since sliced bread. The applications work flawlessly with Apache (if you can use .htaccess in your setup) but take a little tweaking in lighttpd.

Wordpress

I futzed around with this for several hours trying to get this to work properly. Trying to get my head around regular expressions and all the different possible links used in Wordpress. And it all came back to to a really simple lighttpd setting (which oddly enough, doesn’t involve rewite at all).

server.error-handler-404 = "/content/index.php?error=404"

That’s it. That and make sure your permalinks setting doesn’t contain the /index.php/.

Update: The above solution to Wordpress’ permalinks might not be the best. The fact that it’s using the 404 handler might send a 404 response back to the browser. The other issue to worry about is whether or not this is sending a temporary redirect (301). If you have content indexed by a search engine, this will ruin your page ranking. 2008-06-06

Gallery2

Gallery was a bit more difficult. Well, not really. I ended up doing a little R&D (i.e. Rob & Duplicate) from the gallery2 codex.

url.rewrite = (
    "^/(.*)/Rewrite.txt$" => "/$1/Works.txt",
    "^/gallery/v/(\?.+|\ .)?$" => "/gallery/main.php?g2_view=core.ShowItem",
    "^/gallery/admin[/?]*(.*)$" => "/gallery/main.php?g2_view=core.SiteAdmin&$1",
    "^/gallery/d/([0-9]+)-([0-9]+)/([^\/]+)(\?|\ )?(.*)$" =>
    "/gallery/main.php?g2_view=core.DownloadItem&g2_itemId=$1&g2_serialNumber=$2&$3",
    "^/gallery/v/([^?]+)/slideshow.html" =>
    "/gallery/main.php?g2_view=slideshow.Slideshow&g2_path=$1",
    "^/gallery/v/([^?]+)(\?|\ )?(.*)$" =>
    "/gallery/main.php?g2_view=core.ShowItem&g2_path=$1&$3",
    "^/gallery/c/add/([0-9]+).html" =>
    "/gallery/main.php?g2_view=comment.AddComment&g2_itemId=$1",
    "^/gallery/c/view/([0-9]+).html" =>
    "/gallery/main.php?g2_view=comment.ShowAllComments&g2_itemId=$1",
    "^/gallery/p/(.+)" =>
    "/gallery/main.php?g2_controller=permalinks.Redirect&g2_filename=$1"
)

Make sure you change out the ^/gallery/ parts to where you have Gallery2 installed.

Happy Little VPS

All in all, the VPS is running really smoothly. I have everything I had (thechnology wise) at Dreamhost but with twice the performance. My site is defintely faster on the VPS than it was on Dreamhost.

Stay tuned for more HOWTOs on setting up these things on a Gentoo Arch Linux VPS. I plan on writing something for a Postfix/Courier virtual domain setup at some point. If you find this useful, pass it along. I’d be real interested in seeing how this server performs under a heavy load. Perhaps you Digg it?