In moving my website from an Ubuntu server to a virtual host running the TKL Drupal appliance, I decided to keep my conventional development environment i.e. development, staging (test), and production all on the same server.  At some point, I will probably move development and staging to a separate virtual host, but for now I'm keeping them together since that's what I'm familiar with.  Setting up a multi-root installation on a TKL appliance, or Debian/Ubuntu for that matter, requires a few adjustments.  One consideration is how to handle updates.  Debian's Drupal package is setup to support a single-root, multi-site installation.  I want to be proactive on security matters and yet I'm cautious about relying on a packaged update automatically being applied to my production server.  Whenever a new security release of Drupal core or third party module is available, I want to apply it to the staging (test) site first and then run a series of checks before pushing it to the production site.  This requires separate 'roots' for each of the Drupal sites so that the staging site can be upgraded without disturbing the production site.  Fortunately, Drush makes upgrading core and updating modules easy and was designed to support multi-site, multi-root installations.  To make things even easier, I setup site-aliases for each of the three sites, @dev, @test, and @live.  Unfortunately, the TKL installation of Drush includes a shell wrapper for drush which breaks it's multi-root capability.

#!/bin/bash

DRUSH=/usr/local/share/drush/drush
$DRUSH --root=/usr/share/drupal6 $@

This sets the Drupal root to always be /usr/share/drupal6 which can't be overridden by a site alias.

I believe a better way to set the default root for Drush is to put the following in
/etc/drush/drushrc.php

<?php
// $Id: drushrc.php for TurnKey Linux Drupal
// Specify a particular multisite. Change this to your default host name.
$options['l'] = 'http://localhost';

// Specify your Drupal core base directory (useful if you use symlinks).
$options['r'] = '/usr/share/drupal6';

and then link the Drush executable.

ln -s /usr/local/share/drush/drush /usr/local/bin/drush

Drush will then use the default root when none is specified, but it can be overridden by a site-alias or the -r option on the command line.

9. Securing file permissions and ownership

Whenever new modules or updates are downloaded by Drush, the file ownership and permissions are not changed leaving them owned by uid 6226 and readable by the world.  Drupal's Admin Guide recommends securing file permissions and ownership so they are owned by the webmaster and readonly by Apache with the exception of the /files directory which needs to be readwrite by Apache.  Unfortunately, the script provided there doesn't work on TKL Drupal because it doesn't follow symlinks and leaves the third-party modules unprotected.  Hopefully, some future version of Drush will enforce ownership and permissions rules during installation.  In the meantime, I'm working on a modified script that will recognize Drush's site-aliases and follow symlinks in a Debian/Ubuntu/TKL installation.

One question I have for the developers is about the advisibility of creating a separate account for the webmaster role.  Currently all work on the Drupal appliance is done through the root account.  Coming from a RedHat/CentOS background, I'm comfortable working in root and have only been badly burned a couple of times.  It took me awhile to get used to working with Ubuntu, but I came to respect the way it handled security.  Now I'm wondering if it wouldn't be a good idea to add a webmaster account and give it ownership of all the Drupal files.  You've probably already had this debate, but I just had to ask.

Earlier in my testing, I encountered a situation where the Drupal cron.php was being run twice each hour.  I opened a bug report, but then withdrew it after it stopped and I convinced myself it was something I had done.  Yesterday I tripped over the problem again, and was forced to take another look at cron.sh and cron.php.  It began by noticing an unexpected number of e-mails being forwarded by Postfix.  Examining the mail.log showed that every hour the cron daemon was sending a message to the Apache server i.e. www-data@localhost.  That had me puzzled for awhile.  What would be causing cron to send a message, and why to the apache user, www-data, and not to root.  I checked /var/mail/www-data and sure enough there were hundreds of messages from cron.

In Intro to cron, I found the following quote regarding cron and /etc/crontab

MAILTO is who gets mailed the output of each command. If a command cron is 
running has output (e.g. status reports, or errors), cron will email the output 
to whoever is specified in this variable. If no one if specified, then the 
output will be mailed to the owner of the process that produced the output.

I checked the contents of each message and found it was the HTML generated from the Drupal install page.  Next I took a look at the contents of /etc/drupal/6/cron.sh to try and understand what was happening.

#!/bin/sh
# $Id: cron.sh 1878 2008-02-12 10:56:45Z luigi $

for site in /etc/drupal/6/sites/* ; do
	BASE_URL=""

	if [ ! "`basename $site`" = "all" ]; then
		for file in $site/baseurl.php $site/settings.php; do
			[ -f "$file" ] && BASE_URL=`grep '^$base_url' $file | cut -d"'" -f2`
			[ "X$BASE_URL" != "X" ] && break
		done

		if [ "X$BASE_URL" = "X" ] ; then
			BASE_URL='http://localhost'
		fi

		curl --silent --compressed --location $BASE_URL/cron.php
	fi
done

Examining the code shows that cron.sh scans the directory /etc/drupal/6/sites and attempts to run cron.php on every legitimate site it finds, however, if it fails to find a baseurl.php or settings.php with $base_url set, it substitutes 'http://localhost' instead.  In my case, the error was caused by sites/default containing an uninitialized settings.php.  If 'localhost' was a site alias for my main site, the result would be that cron ran twice each hour.  In my case, Drupal found the uninitialized settings.php, assumed it had never been run before and launched install.php instead of cron.php.  The resulting output from curl was then e-mailed to www-data.  Note that the 'bad' behavior can be triggered any unused directory under /etc/drupal/6/sites/ even an empty directory.

I have racked my brain, but I can't think of a single reason why you would want to run cron.php on a site without a properly initialized settings.php.  Here is the modified version that I think makes more sense.

#!/bin/sh
# $Id: cron.sh modified for TurnKey Drupal Appliance $

for site in /etc/drupal/6/sites/* ; do
	BASE_URL=""

	if [ ! "`basename $site`" = "all" ]; then
		for file in $site/baseurl.php $site/settings.php; do
			[ -f "$file" ] && BASE_URL=`grep '^$base_url' $file | cut -d"'" -f2`
			[ "X$BASE_URL" != "X" ] && break
		done

		if [ "X$BASE_URL" != "X" ]; then
		        curl --silent --compressed --location $BASE_URL/cron.php
		fi
	fi
done

Note that it runs cron.php only if it finds $base_url set in either baseurl.php or settings.php.  I also changed /etc/crontab and added 'MAILTO=root' so that all future cron errors will be mailed to root (or the alias set in step 2) instead of ending up in an unread mailbox.

10. Move CKeditor to sites/all/libraries/ckeditor

Currently the CKeditor is downloaded to a subdirectory sites/all/modules/ckeditor/ckeditor.  This has the disadvantage of being overwritten when a new version of ckeditor is installed.  A Drupal best practice that I try to follow it to place all third party packages that can't be included because of licensing concerns into sites/all/libraries.  This includes the gmap software.  The CKeditor module will automatically find the javascript in sites/all/libraries/ckeditor.

In the interest of saving time I'm going to post a finalized version of apache.conf for Drupal6.  Most of the suggested changes come from the series of articles;

Drupal and Apache Security Checklist Part 1, Part 2, Part 3

by Nadeau Software Consulting

# Moved to httpd.conf so drupal6 can be disabled without affecting other virtual hosts.
# NameVirtualHost *:80
# NameVirtualHost *:443

<VirtualHost *:80>
    UseCanonicalName Off
    ServerName drupal6
    ServerAdmin  webmaster@localhost
    DocumentRoot /usr/share/drupal6/
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/cert.pem
    ServerAdmin  webmaster@localhost
    DocumentRoot /usr/share/drupal6/
</VirtualHost>

<DirectoryMatch "/usr/share/drupal6/(?!(.+/)+)">
# Block any file that starts with "."
    <FilesMatch "^\..*$">
        Order allow,deny
    </FilesMatch>

# Block all files with "." in their names
    <FilesMatch "^.*\..*$">
        Order allow,deny
    </FilesMatch>

# Allow "." files with safe content types
    <FilesMatch "^.*\.(css|html?|txt|js|xml|xsl|gif|ico|jpe?g|png)$">
        Order deny,allow
    </FilesMatch>
    <FilesMatch "^.*\.(f4a|f4b|f4p|f4v|flac|flv|mov|mp3|qt|swf)$">
        Order deny,allow
    </FilesMatch>
    <FilesMatch "^.*\.(docx?|pptx?|xlsx?|odt|ods|odp|pdf)$">
        Order deny,allow
    </FilesMatch>

# Allow from anywhere
    <FilesMatch "^(index|xmlrpc).php$">
        Order deny,allow
    </FilesMatch>

# Allow access to cron from server and local network only
    <FilesMatch "^cron.php$">
        Order allow,deny
        Allow from 127.0.0.1   # Localhost
        Allow from 192.168.    # Local network (use your local IP range)
    </FilesMatch>

# Allow access to install and update from server and local network only
    <FilesMatch "^(install|update).php$">
        Order allow,deny
        Allow from 127.0.0.1   # Localhost
        Allow from 192.168.    # Local network (use your local IP range)
    </FilesMatch>

</DirectoryMatch>

<Directory /usr/share/drupal6/>
    Options +FollowSymLinks
    AllowOverride All
    order allow,deny
    allow from all
</Directory>

# Include rewrite rules here instead of .htaccess so it will be easier
# to disable .htaccess during performance tuning.
<IfModule mod_rewrite.c>
    RewriteEngine on

# If your site can be accessed both with and without the 'www.' prefix, you
# can use one of the following settings to redirect users to your preferred
# URL, either WITH or WITHOUT the 'www.' prefix. Choose ONLY one option:
#
# To redirect all users to access the site WITH the 'www.' prefix,
# (http://example.com/... will be redirected to http://www.example.com/...)
# adapt and uncomment the following:
# RewriteCond %{HTTP_HOST} ^example\.com$ [NC]
# RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]
#
# To redirect all users to access the site WITHOUT the 'www.' prefix,
# (http://www.example.com/... will be redirected to http://example.com/...)
# uncomment and adapt the following:
# RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
# RewriteRule ^(.*)$ http://example.com/$1 [L,R=301]

# Return 404's for any access to CSS, JS, and images
# unless they come from your site's pages or a trusted host.
    RewriteCond %{REMOTE_ADDR}      !^127.0.0.1$
    RewriteCond %{HTTP_REFERER}     !^https?://%{HTTP_HOST}/.*$ [NC]
    RewriteRule .(css|js|ico|gif|jpe?g|png)$        - [R=404]

# Return 404's for all Drupal .php, .inc, .module, and .info files
# if served to hosts other than the local network hosts, but allow
# /index.php and /xmlrpc.php.
    RewriteCond %{REMOTE_ADDR}      !^127.0.0.1$
    RewriteCond %{REQUEST_URI}      ^.*\.(php|inc|module|info)$
    RewriteCond %{REQUEST_URI}      !^/(index|xmlrpc).php$
    RewriteRule .*  - [R=404]

# Rewrite index.php to /
    RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*index\.php\ HTTP/
    RewriteRule ^(.*)index\.php$ /$1 [L,R=301]

# Force trailing / for directories
    RewriteCond    %{REQUEST_FILENAME}  -d
    RewriteRule    ^(.+[^/])$           $1/  [L,R=301]

</IfModule>

The last two items were suggested by Drupal SEO: How Duplicate Content Hurts Drupal Sites and Duplicate Title Tags "/" and "/index.php"  

Speaking of SEO, you might want to consider adding Path Redirect and Global Redirect modules.  I found that both were necessary to deal with SEO issues related to PathAuto.

Alon,  Thanks for taking the time to consider all these changes.  I know there are a lot of them.

As I continue to review my notes and watch my server logs, I'm finding additional changes that I've made.

12. Increase php's 'memory_limit' from 32MB to 64MB

As more modules are added, Drush appears to require more memory for php scripts.
Change the value of 'memory_limit' in /etc/php5/cli/php.ini from 32MB to 64MB.  I choose to also increase the limit in /etc/php5/apache2/php.ini.

13. Move getid3 to third-party folder and fix Permissions

Even though the getID3 module is disabled, it appears to be accessed by admin resulting in errors in the logs.  The files in /usr/share/php-getid3 are owned by root and apache lacks read permission.

opendir(sites/all/modules/getid3/getid3/getid3): failed to open dir: Permission denied in /var/www/drupal/includes/file.inc on line 931.

Move the location of the getID3 third-party download to sites/all/libraries for consistency and so it doesn't get overwritten when the module is updated.

From the drupal root directory,

   # mv sites/all/modules/getid3/getid3 sites/all/libraries

Change ownership so apache has read access
   # chgrp -R www-data /usr/share/php-getid3