[Update 2] – It seems W3 Total Cache support has been silently discontinued – it’s not been updated for over 12 months now, and has allegedly got some security issues. Because of this, I would recommend not installing W3 Total Cache as part of these instructions, everything else should run fine. [/Update]
[Update] – I’ve tested this using the new Ubuntu 12.04 LTS Edition, and the settings all appear to work correctly without modification. Let me know if you find a problem [/Update]
[Update September 2013] – I’ve recently moved my own blog hosting to a $10 a month VPS from Digital Ocean (that link gets you a $10 discount via an affiliate link) and have been very happy with them, including the performance of the blog.
These instructions are the rather verbose, but hopefully easy enough to follow, steps to build a new Linux server using Varnish, Nginx, W3 Total Cache, and WordPress, to build a WordPress blog on a Amazon Micro server (or equivalent), all costing under $15 a month, capable of sustaining 10 million hits per day, as measured by blitz.io.
10 Million hits per day with WordPress on a $15 virtual server
Install Ubuntu 11.10 (Oneiric) on a new virtual private server- it needs to be 11.10 for all the packages that I’m going to list to work out of the box, but it’s all possible to do with other distributions.
I used Amazon EC2 to build my test server, but Linode are also very good. As of September 2013, this blog is now hosted on Digital Ocean
For the purpose of the documentation, my server details were as follows, yours will be different:
Public DNS Name: ec2-23-20-235-223.compute-1.amazonaws.com
Public IP Address: 23.20.235.223
Login to the server and become root
login as ubuntu via ssh, then run the sudo -i command to become root
ssh ubuntu@ec2-23-20-235-223.compute-1.amazonaws.com
sudo -i
Configure a firewall first
Since we’re going to be installing various network services which by default listen on all interfaces, it’s important to configure a firewall.
For Ubuntu, this is nice and easy, simply use the ufw package.
Execute the following commands to configure it:
ufw allow ssh
ufw allow http
ufw logging off
ufw enable
Once this is done, your server has a relatively secure firewall, though it’s worth looking at fail2ban to prevent brute force password attacks.
If you’re using Amazon EC2, you’ll also need to open the Security Group to allow traffic on port 80. You can do this using the AWS Security Groups Console, you might need to change the region. Select the security group you used when you started the instance, and then click “Inbound”, then select “HTTP” from the drop down menu, then finally click “Add Rule”. You don’t need to restart the instance for it to take effect.
Install and Configure MySQL
apt-get update
apt-get install mysql-server
When prompted, set a mysql “root” user password
mysql -u root -p
When prompted, enter your newly set root password
At the mysql> prompt, run the following 4 commands, replacing ENTER_A_PASSWORD with a password of your own
CREATE DATABASE wordpress;
GRANT ALL PRIVILEGES ON wordpress.* TO “wp_user”@”localhost” IDENTIFIED BY “ENTER_A_PASSWORD”;
FLUSH PRIVILEGES;
EXIT
That’s MySQL installed, ready for the PHP and Web server installation (nginx).
Install and configure PHP
We need to install not just PHP, but the PHP FPM system, APC, and the MySQL module
apt-get install php5-fpm php-pear php5-common php5-mysql php-apc php5-gd
Edit /etc/php5/fpm/php.ini and add these lines at the bottom:
[apc]
apc.write_lock = 1
apc.slam_defense = 0
Edit /etc/php5/fpm/pool.d/www.conf
Replace
listen = 127.0.0.1:9000
with
listen = /dev/shm/php-fpm-www.sock
Below that, insert these 3 lines
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
Then, further down in the same file, replace these 2 lines
user = www-data
group = www-data
with
user = nginx
group = nginx
Save the file, PHP FPM is now complete, but it won’t work until we install nginx, so don’t worry about starting it now.
Install and Configure Nginx
Instructions based on the Nginx website.
Download the nginx secure key to verify the package
cd /tmp/
wget http://nginx.org/keys/nginx_signing.key
apt-key add /tmp/nginx_signing.key
Add the sources to the APT sources file by running these 2 commands (the >> is important!)
echo “deb http://nginx.org/packages/ubuntu/ lucid nginx” >> /etc/apt/sources.list
echo “deb-src http://nginx.org/packages/ubuntu/ lucid nginx” >> /etc/apt/sources.list
Download and install nginx by running
apt-get update
apt-get install nginx
When that completes, nginx will be installed, but needs configuring for WordPress.
nginx configuration files are in /etc/nginx
First, edit /etc/nginx/nginx.conf
Inside the http section , insert the following lines so that when you later add varnish in front, things don’t break all over the place, and so varnish caches compressed files:
port_in_redirect off;
gzip on;
gzip_types text/css text/xml text/javascript application/x-javascript;
gzip_vary on;
Next, cd to /etc/nginx/conf.d and create a new file, /etc/nginx/conf.d/drop with the contents of the drop file from GitHub
Then, replace /etc/nginx/conf.d/default.conf with the contents of the github default.conf file changing all entries for domainname.com with your own domain name (there’s 3 entries, including 1 near the bottom)
Make a directory, /var/www/ and set the ownership:
mkdir -p /var/www/
chown nginx:nginx /var/www/
chmod 775 /var/www
That’s nginx configured, restart it and the PHP FPM service by running:
service nginx restart
service php5-fpm restart
Now, you’re actually ready to install WordPress!
This is pretty simple, run:
cd /tmp
wget http://wordpress.org/latest.tar.gz
tar zxvf latest.tar.gz
cd wordpress
mv * /var/www/
chown -R nginx:nginx /var/www
To configure WordPress, run:
cp /var/www/wp-config-sample.php /var/www/wp-config.php
chown nginx:nginx /var/www/wp-config.php
In a web browser, visit https://api.wordpress.org/secret-key/1.1/salt/ and copy the results
Edit /var/www/wp-config.php
and scroll down to fine the AUTH_KEY line down to NONCE_SALT, and replace them with the values you copied from the api.wordpress.org site
Then, replace the default values with the MySQL ones you chose earlier (not the root user):
define(‘DB_NAME’, ‘database_name_here’);
define(‘DB_USER’, ‘username_here’);
define(‘DB_PASSWORD’, ‘password_here’);
And once it’s done, if you’ve not had any errors and your domain name is pointing at the right IP (this is important!), then you’ll be able to visit your domain and see the WordPress configuration page at http://www.domainname.com/wp-admin/install.php
Go through install questions, choose a sensible username and password (it’s highly recommended you change the admin user from “admin” to something else.
Go to settings then select permalinks, and choose “Custom Structure”, and paste in the value below (including the % symbols)
/%post_id%/%postname%
Then hit “Save Changes”
It’s time to create a test post, so click on “Posts” then “Add New”
Enter a title and body, then hit “Post”, and make a note of it’s friendly URL.
Run a blitz.io test to see how we’re doing
100 users, 60 seconds. Timeouts, low hit rates, errors, etc. CPU flat out, seems to be the initial bottleneck.
This rush generated 632 successful hits in 1.0 min and we transferred 1.76 MB of data in and out of your app. The average hit rate of 9.81/second translates to about 847,776 hits/day.
You got bigger problems though: 34.91% of the users during this rush experienced timeouts or errors!
So the server is running, but it’s still too slow!
Next, we will enable the WordPress caching systems
Go to the wordpress admin page, then plugins, and click install new plugin.
Update Skip the W3 Total Cache plugin if you’re following this as a tutorial, it’s not been updated in ages. I’ll update this post in the future with W3 Super Cache instructions instead, but everything else should work as it is for now
Search for “W3 Total Cache”, then click “Install Now” when the search results return. When installation is complete, click “Activate Plugin”.
Go to the new “Performance” section in the menu at the left side of the page.
Scroll through the cache options, selecting “PHP APC” at each opportunity and enabling the following 2 sections:
Database Cache
Object Cache
Hit “Save All Settings” then hit “Deploy”
Rerun the blitz.io test again, performance should be much improved
This rush generated 2,902 successful hits in 1.0 min and we transferred 27.59 MB of data in and out of your app. The average hit rate of 46/second translates to about 4,006,008 hits/day.
If we then up the blitz.io run from 100 users to 250 users, there are still problems we need to fix:
This rush generated 4,733 successful hits in 1.0 min and we transferred 36.20 MB of data in and out of your app. The average hit rate of 72/second translates to about 6,297,653 hits/day.
You got bigger problems though: 5.49% of the users during this rush experienced timeouts or errors!
You can see the Blitz results from the Nginx And APC configuration in this PDF.
So still not perfect – Time to install varnish 3!
apt-get install varnish
Edit /etc/varnish/default.vcl replace the contents with the file default.vcl file from github.
Edit /etc/default/varnish
Change the section
DAEMON_OPTS=”-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m”
to
DAEMON_OPTS=”-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,64m”
Next, we need to edit the nginx configuration to listen on port 8080, instead of port 80 – Varnish is going to be running on port 80 instead.
Edit /etc/nginx/conf.d/default.conf and replace
listen 80;
with
listen 8080;
Save the file, then run
service nginx restart
service varnish restart
Re-run the blitz.io test with 100 users
Almost no CPU usage, should get perfect results
This rush generated 2,959 successful hits in 1.0 min and we transferred 28.36 MB of data in and out of your app. The average hit rate of 47/second translates to about 4,093,000 hits/day.
Re-run the blitz with 250 users
This rush generated 7,342 successful hits in 1.0 min and we transferred 70.38 MB of data in and out of your app. The average hit rate of 117/second translates to about 10,134,627 hits/day.
You can see the full final run blitz performance details in this PDF.
There it is, 10 million hits per day using WordPress on a box costing less than $20 a month, all thanks to varnish and nginx, easy!
Remember to get your $10 discount from Digital Ocean.
I’ve previously blogged about my belief that ARM servers are a big part of the future, and that server manufacturers are missing a trick by not testing the market with small, cheap, ARM servers, but I wondered if maybe they knew something I didn’t.
