[Update 2] – It seems W3 Total Cache support has been silently discontinued – it’s not been updated for over 12 months now, and has allegedly got some security issues. Because of this, I would recommend not installing W3 Total Cache as part of these instructions, everything else should run fine. [/Update]
[Update] – I’ve tested this using the new Ubuntu 12.04 LTS Edition, and the settings all appear to work correctly without modification. Let me know if you find a problem [/Update]
[Update September 2013] – I’ve recently moved my own blog hosting to a $10 a month VPS from Digital Ocean (that link gets you a $10 discount via an affiliate link) and have been very happy with them, including the performance of the blog.
These instructions are the rather verbose, but hopefully easy enough to follow, steps to build a new Linux server using Varnish, Nginx, W3 Total Cache, and WordPress, to build a WordPress blog on a Amazon Micro server (or equivalent), all costing under $15 a month, capable of sustaining 10 million hits per day, as measured by blitz.io.
10 Million hits per day with WordPress on a $15 virtual server
Install Ubuntu 11.10 (Oneiric) on a new virtual private server- it needs to be 11.10 for all the packages that I’m going to list to work out of the box, but it’s all possible to do with other distributions.
I used Amazon EC2 to build my test server, but Linode are also very good. As of September 2013, this blog is now hosted on Digital Ocean
For the purpose of the documentation, my server details were as follows, yours will be different:
Public DNS Name: ec2-23-20-235-223.compute-1.amazonaws.com
Public IP Address: 23.20.235.223
Login to the server and become root
login as ubuntu via ssh, then run the sudo -i command to become root
ssh ubuntu@ec2-23-20-235-223.compute-1.amazonaws.com
sudo -i
Configure a firewall first
Since we’re going to be installing various network services which by default listen on all interfaces, it’s important to configure a firewall.
For Ubuntu, this is nice and easy, simply use the ufw package.
Execute the following commands to configure it:
ufw allow ssh
ufw allow http
ufw logging off
ufw enable
Once this is done, your server has a relatively secure firewall, though it’s worth looking at fail2ban to prevent brute force password attacks.
If you’re using Amazon EC2, you’ll also need to open the Security Group to allow traffic on port 80. You can do this using the AWS Security Groups Console, you might need to change the region. Select the security group you used when you started the instance, and then click “Inbound”, then select “HTTP” from the drop down menu, then finally click “Add Rule”. You don’t need to restart the instance for it to take effect.
Install and Configure MySQL
apt-get update
apt-get install mysql-server
When prompted, set a mysql “root” user password
mysql -u root -p
When prompted, enter your newly set root password
At the mysql> prompt, run the following 4 commands, replacing ENTER_A_PASSWORD with a password of your own
CREATE DATABASE wordpress;
GRANT ALL PRIVILEGES ON wordpress.* TO “wp_user”@”localhost” IDENTIFIED BY “ENTER_A_PASSWORD”;
FLUSH PRIVILEGES;
EXIT
That’s MySQL installed, ready for the PHP and Web server installation (nginx).
Install and configure PHP
We need to install not just PHP, but the PHP FPM system, APC, and the MySQL module
apt-get install php5-fpm php-pear php5-common php5-mysql php-apc php5-gd
Edit /etc/php5/fpm/php.ini and add these lines at the bottom:
[apc]
apc.write_lock = 1
apc.slam_defense = 0
Edit /etc/php5/fpm/pool.d/www.conf
Replace
listen = 127.0.0.1:9000
with
listen = /dev/shm/php-fpm-www.sock
Below that, insert these 3 lines
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
Then, further down in the same file, replace these 2 lines
user = www-data
group = www-data
with
user = nginx
group = nginx
Save the file, PHP FPM is now complete, but it won’t work until we install nginx, so don’t worry about starting it now.
Install and Configure Nginx
Instructions based on the Nginx website.
Download the nginx secure key to verify the package
cd /tmp/
wget http://nginx.org/keys/nginx_signing.key
apt-key add /tmp/nginx_signing.key
Add the sources to the APT sources file by running these 2 commands (the >> is important!)
echo “deb http://nginx.org/packages/ubuntu/ lucid nginx” >> /etc/apt/sources.list
echo “deb-src http://nginx.org/packages/ubuntu/ lucid nginx” >> /etc/apt/sources.list
Download and install nginx by running
apt-get update
apt-get install nginx
When that completes, nginx will be installed, but needs configuring for WordPress.
nginx configuration files are in /etc/nginx
First, edit /etc/nginx/nginx.conf
Inside the http section , insert the following lines so that when you later add varnish in front, things don’t break all over the place, and so varnish caches compressed files:
port_in_redirect off;
gzip on;
gzip_types text/css text/xml text/javascript application/x-javascript;
gzip_vary on;
Next, cd to /etc/nginx/conf.d and create a new file, /etc/nginx/conf.d/drop with the contents of the drop file from GitHub
Then, replace /etc/nginx/conf.d/default.conf with the contents of the github default.conf file changing all entries for domainname.com with your own domain name (there’s 3 entries, including 1 near the bottom)
Make a directory, /var/www/ and set the ownership:
mkdir -p /var/www/
chown nginx:nginx /var/www/
chmod 775 /var/www
That’s nginx configured, restart it and the PHP FPM service by running:
service nginx restart
service php5-fpm restart
Now, you’re actually ready to install WordPress!
This is pretty simple, run:
cd /tmp
wget http://wordpress.org/latest.tar.gz
tar zxvf latest.tar.gz
cd wordpress
mv * /var/www/
chown -R nginx:nginx /var/www
To configure WordPress, run:
cp /var/www/wp-config-sample.php /var/www/wp-config.php
chown nginx:nginx /var/www/wp-config.php
In a web browser, visit https://api.wordpress.org/secret-key/1.1/salt/ and copy the results
Edit /var/www/wp-config.php
and scroll down to fine the AUTH_KEY line down to NONCE_SALT, and replace them with the values you copied from the api.wordpress.org site
Then, replace the default values with the MySQL ones you chose earlier (not the root user):
define(‘DB_NAME’, ‘database_name_here’);
define(‘DB_USER’, ‘username_here’);
define(‘DB_PASSWORD’, ‘password_here’);
And once it’s done, if you’ve not had any errors and your domain name is pointing at the right IP (this is important!), then you’ll be able to visit your domain and see the WordPress configuration page at http://www.domainname.com/wp-admin/install.php
Go through install questions, choose a sensible username and password (it’s highly recommended you change the admin user from “admin” to something else.
Go to settings then select permalinks, and choose “Custom Structure”, and paste in the value below (including the % symbols)
/%post_id%/%postname%
Then hit “Save Changes”
It’s time to create a test post, so click on “Posts” then “Add New”
Enter a title and body, then hit “Post”, and make a note of it’s friendly URL.
Run a blitz.io test to see how we’re doing
100 users, 60 seconds. Timeouts, low hit rates, errors, etc. CPU flat out, seems to be the initial bottleneck.
This rush generated 632 successful hits in 1.0 min and we transferred 1.76 MB of data in and out of your app. The average hit rate of 9.81/second translates to about 847,776 hits/day.
You got bigger problems though: 34.91% of the users during this rush experienced timeouts or errors!
So the server is running, but it’s still too slow!
Next, we will enable the WordPress caching systems
Go to the wordpress admin page, then plugins, and click install new plugin.
Update Skip the W3 Total Cache plugin if you’re following this as a tutorial, it’s not been updated in ages. I’ll update this post in the future with W3 Super Cache instructions instead, but everything else should work as it is for now
Search for “W3 Total Cache”, then click “Install Now” when the search results return. When installation is complete, click “Activate Plugin”.
Go to the new “Performance” section in the menu at the left side of the page.
Scroll through the cache options, selecting “PHP APC” at each opportunity and enabling the following 2 sections:
Database Cache
Object Cache
Hit “Save All Settings” then hit “Deploy”
Rerun the blitz.io test again, performance should be much improved
This rush generated 2,902 successful hits in 1.0 min and we transferred 27.59 MB of data in and out of your app. The average hit rate of 46/second translates to about 4,006,008 hits/day.
If we then up the blitz.io run from 100 users to 250 users, there are still problems we need to fix:
This rush generated 4,733 successful hits in 1.0 min and we transferred 36.20 MB of data in and out of your app. The average hit rate of 72/second translates to about 6,297,653 hits/day.
You got bigger problems though: 5.49% of the users during this rush experienced timeouts or errors!
You can see the Blitz results from the Nginx And APC configuration in this PDF.
So still not perfect – Time to install varnish 3!
apt-get install varnish
Edit /etc/varnish/default.vcl replace the contents with the file default.vcl file from github.
Edit /etc/default/varnish
Change the section
DAEMON_OPTS=”-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m”
to
DAEMON_OPTS=”-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,64m”
Next, we need to edit the nginx configuration to listen on port 8080, instead of port 80 – Varnish is going to be running on port 80 instead.
Edit /etc/nginx/conf.d/default.conf and replace
listen 80;
with
listen 8080;
Save the file, then run
service nginx restart
service varnish restart
Re-run the blitz.io test with 100 users
Almost no CPU usage, should get perfect results
This rush generated 2,959 successful hits in 1.0 min and we transferred 28.36 MB of data in and out of your app. The average hit rate of 47/second translates to about 4,093,000 hits/day.
Re-run the blitz with 250 users
This rush generated 7,342 successful hits in 1.0 min and we transferred 70.38 MB of data in and out of your app. The average hit rate of 117/second translates to about 10,134,627 hits/day.
You can see the full final run blitz performance details in this PDF.
There it is, 10 million hits per day using WordPress on a box costing less than $20 a month, all thanks to varnish and nginx, easy!
Remember to get your $10 discount from Digital Ocean.
Impressive! Now, can you show me how I can get 10 million (genuine!) hits a day? 😉
Sure, just transfer £5 million into my bank account, and I’ll send you the details 🙂
Make a page with 1,000,000 1px icons on it and have 10 of your friends go to it. You could do that with $5 godaddy hosting. Hits != page views. This article must be misleading to the unaware.
Wasn’t expecting this article
Rush, if you can think of any off the top of your head, can you point me in the direction of some resources that elaborate on your statement? I’d rather not be “the unaware.” Is it as simple as hits = requests to the server, and there are may requests in a given page load? I’ve always equated hits & page views until I read your statement.
Yes you are correct. Wrote a quick example: http://www.rushfrisby.com/hits/
Thanks, good to know!
My comment must be misleading for those lacking a sense of humour.
I’m glad you said that. I was just about to post the same thing. Page views and hits are not the same.
For any of you on shared hosting you can verify this by checking your modlogan stats summary page. Total hits and Total pages.Same thing in Webalizer. You will see a huge difference in the numbers.
I don’t care for either one of those stats trackers as they both count search engines and other non-users as “hits” and “page views”. But if you take the time to go through them you can eliminate bots and eliminate all visits you yourself made(by IP) you can get a pretty good idea of where your site is. Actually better than Googles analytics, as Google doesn’t always show you everything. But using these with GA will really give you the full picture traffic to your site.
At least in the context of blitz.io stats (I’m the architect for blitz), hits is very well equal to page views. For one thing, blitz doesn’t fetch static asserts (right now) and focus purely on server-side performance. So as for the stats we measure, hits and page views are exactly the same.- @k0ws1k:twitter
What about adding cloudflare? Those micro’s do not have great performance, i doubt you will get that throughput sustained. Amazon will start to steal CPU away from that micro, they do not have a dedicated core. What was your CPU performance during that?
This blog actually runs behind CloudFlare, and it’s pretty amazing yeah. The only issue with it is that debugging what’s happening behing cloudflare can be a little confusing.
For CPU performance, it was ticking over – between 5-10% CPU usage at the very peak. Network bandwidth and my account with blitz.io seemed to be the real limiting factor.
Interesting, you may be able to sustain it if it’s only 5-10% cpu. Another question, I wonder how it performs compared to $3-$4 a month shared hosting.
Varnish takes CPU out of the equation.
Great post!
I am amazed at what Varnish can do. It’s simply the best out there.
Have a good one,
Kent
i can barely load this blog post. so i guess you are not putting this info into use! 🙂
Odd, sorry about that, loading fine here!
Can I ask where in the world you are? The server is in the UK
Loading perfectly here in the U.S., thank you.
Pingback: 10 Million hits a day with Wordpress using a $15 server - Monday By Noon
10 million hits a day for this blog. Are u kidding me? You might be able to have a cheap workable server, but that much hits for this blog sounds just hilarious. Is the title of the post just to fetch attention ?
I don’t expect to ever get 10 million hits a day (or even 10k!) on this blog, it’s simply an experiment to see how far you can push wordpress on a small server.
As it it, the results work well, and do (I think) back up the title, but I’m sorry if you some how feel mislead.
Not good enough ewan, I need you to explain to me how to write a blog that will get 10 million hits a day. Im thinking it should be about cats…
I got here by googling “cats”, but all I found was this comment.
The first paragraph clearly states it’s 10 million from blitz.io – durr.
Did you actually read the contents of the article or just the title?
Pingback: Setting Up WordPress for 10 Million Hits a Day | Faraway, So Close
Good to know. I wish one of my sites would get 10 million views a day!
May I ask how much RAM your EC2 instance has?
Amazon EC2 Micro has 613MB memory.
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/instance-types.html
Pingback: [GNU/Linux] varnish nginx php-fpm mysql
Great article: very descriptive. My approach was to do all the plugin configuration through nginx configuration, and to put the site behind CloudFlare for geo-optimization and image/cdn advantages:
http://danielmiessler.com/blog/how-to-run-a-wicked-fast-wordpress-instance
CloudFlare won’t scale as far as a good plugin, and isn’t nearly as configurable. You can barely control anything with CloudFlare.
The caching stuff you do in nginx before CloudFlare is involved. So, nginx configuration is the substitution for the plugin, not CloudFlare. CloudFlare gives you the ability to get your static content served from the closest geographical location possible, etc.
Cloudflare is fantastic yes, this blog sits behind it to offload static content, and handles things like some simple hacking attacks, which is great.
What is this “Virtualisation” your blog header speaks of? Now serving 10 million misspellings per day.
Get back to the USA.
It’s correct also; the British spelling.
Perhaps I should try this out with Amazon Linux AMI. I honestly don’t believe this will get 10 million hits a day, but it will still be beefy.
I’m curious to see how much RAM was used according to the AWS docs (http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/instance-types.html) Mico’s only have 613MB and to my limited knowledge of how nginx works it’s a memory hog, using lots of RAM, which is what makes it “fast”.
Also according to the AWS docs (note the graphs for micro’s) http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html if they (AWS) see youre Micro spiking for long periods of time they will throttle it, hence slowing it down, so the only way around that would be to load balance multiple Micro’s.
How did you configure MySQL to handle X number of connections?
Micro’s also have no root storage their local storage is 7.9G, but isn’t reliable for persistent data.
Is blitz like Apache AB testing? Haven’t used blitz before.
None the less I found your post interesting and its topics like this that are/will truly push what AWS has to offer for creative users. 😀
Sorry just want to correct myself you’ll have the same issue with load balancing micro’s cause they aren’t/can’t be used for consistent cpu usage.
The memory of the micro server was pretty much fully consumed, otherwise I’d of made the Varnish cache size bigger if anything.
The CPU of the server is pretty idle, so in theory, they shouldn’t be throttling it, but you never know.
The main thing to do to speed up WordPress is to make sure it almost never talks to the database, so this is running with the default MySQL configuration from Ubuntu 11 – nothing was changed that’s not on the blog post itself.
This was done using a micro server backed by persistent EBS – AMI image ami-baba68d3 if you want to try it for yourself.
Actually Nginx is *not* a memory hog, which is part of what makes it fast (leaves memory for apps and VM). If I had to guess at Ewin’s memory utilization, I’d venture most of it is used by (in descending order):
1) Varnish
2) PHP/Wordpress
3) MySQL
4) Nginx
@Ewin: care to share the numbers with us?
Sorry I’ve not kept accurate measurements, but you’re right, it’s Varnish and PHP at the top, MySQL, then nginx.
PHP could well be the top consumer, mostly because it’s launching too many children for what’s really needed, it could be reduced I’m sure.
I think you’d do better without varnish; it really isn’t very good and gets worse under load. Nginx by itself can easily outstrip it (configured as a caching proxy and web server) with less memory and CPU.
If you really want serious performance out of small hardware, you should take a look at GWAN: http://gwan.com/ (they have some interesting benchmarks on other combinations too)
Pingback: 10 Million hits a day with Wordpress using a $15 server | Ewan's … | WordPress Planet
What invalidates the Varnish cache?
The -t switch in the varnish file is a timeout, after that it’ll expire content automatically. I’ve not gone through the configuration here to do a proper varnish to wordpress cache invalidation, I think it’d be a whole post on it’s own.
I believe W3 Total Cache can handle Varnish invalidation for you these days.
You’re right yes, and my varnish config file supports it, I just didn’t mention it configuring it in WordPress in the article – I probably should have judging by the comments on it 🙂
Pingback: Dougal Campbell: 10 Million hits a day with WordPress using a $15 server | Ewan’s Blog on IT and stuff like it - All News & Campaigns.
Any pointers on how to do this when the blog is in a subdomain (e.g. blog.domain.com)? I’ve followed all the steps successfully until the WordPress installation. the install.php file gives me a 404. The domain name resolves to the right IP and going to the root directory of the subdomain gives a “Error Establishing Database Connection”.
Thanks for the post. It’s awesome!
That’s odd, I’d start by looking in /var/log/nginx/error.log
Make sure that the site name in the nginx default.conf is set to blog.domain.com instead of domain.com
How are you managing to get as low as 117/sec? You can easily get 2,000 a sec on Linode with most decent frameworks if you cache output :S
For those who are looking for an alternative setup to hit 10 million hits/day (in a demo or lightweight live WordPress environment using blitz with 250 users in 60 seconds), the following would do the same…
1. Varnish + Apache
2. Nginx + any backend (Apache/PHP-FPM) + Memcached + Batcache
3. Nginx + any backend (Apache/PHP-FPM) + Memcached + WP-FFPC
4. Nginx + Nginx proxy cache + any backend (Apache/PHP-FPM)
Each setup works great on a demo or minimal live WordPress installation. Beyond that, each has its own advantage and disadvantage. Also, there are literally hundreds of ways to tweak this set up depending on the requirement.
Thanks for the tip on WP-FFPC, it looks brand new and supports memcached. I’ve been with WPSC for ages, and W3TC has too many performance issues, so I really wanted WPSC with memcached support. This may just be it, will definitely try it.
Glad to share it. I’ve been using WP-FFPC for sometime in a demo environment. It works great on Ubuntu, but couldn’t make it to work on Amazon Linux without a bug that couldn’t be fixed yet. It displays a random 4-digit alphanumeric number on top-left, during the first page view at random that is annoying at times. However, further views are displaying correctly. If you or anyone could find a fix, I’m all ears.
i cant understand all this. Please help me how can i get 10 million hits/day on my website
Can you give some specifics on what performance issues you’ve seen with W3TC?
Curious, what issues have you had / seen?
Frederick, we have this conversation from time to time 🙂 Last time was with my Twitter account @ArtemR, after which I sent you a very detailed email with a history of communication with your the observations. I never heard back.
Wow, thanks for spreading the word on my plugin! 🙂
I will try this setup! Thanks!
Hi Ewan, I had a crack at your instructions and got it going on http://ec2-107-22-55-190.compute-1.amazonaws.com/
Thanks for the detailed how-to! I have been looking for something as detailed as this for a while. You are a genius!
The only thing I haven’t got working is my Route53 recordset on http://www.mjsweet.com, hopefully it will finish propagating soon.
Also, I noticed that WordPress isn’t crunching thumbs on this install, does it have to do with permissions? Also minify doesn’t seem to be working too, I presume for the same reason? I’m pretty sure I got everything right on the install.
You’ve not done anything wrong, I just didn’t cover it for simplicity – tweaking these things can be as long a set of instructions as this whole post 🙂
You’ll need to add a rewrite rule to the nginx default.conf file (and while you’re at it, the default.conf file on github has a minor fix that you’ll probably need too).Have a look at this page for fully configuring W3 Total Cache and Nginx http://elivz.com/blog/single/wordpress_with_w3tc_on_nginx/
Thanks again Ewan, that will help alot – also fixes Minify
Hey Ewan, I fixed it… turns out I didn’t install GD so PHP wasn’t resizing the images at all.
Just did this and then restarted nginx and php:
apt-get install php5-gd
While I remember, you might know this but others won’t.
While waiting for DNS to propagate, you can add the following line to your hosts file to test from your local PC immediately:
107.22.55.190 http://www.mjsweet.com
The hosts file on Windows is c:windowssystem32driversetchosts (no file extension).
Then, if you want to run a blitz test, run it with these parameters, changing Host for your new hostname, and the URL for public domain or IP of your new server:
-T 5000 -p 1-50:60 -H 'Host: www.ewan.im' http://ec2-23-20-148-209.compute-1.amazonaws.com/4/this-is-the-test
That way, blitz will include the new hostname in the header, even though it can’t resolve it yet.
Pingback: 10 Million hits a day with Wordpress using a $15 server | Ewan’s Blog on IT and stuff like it | WordBuff.in
Does blitz.io hit the same page every time? Does it load resources (images, css, js, etc)? Real-world performance could be way different, depending on what your claim entails. Add blogs without JS comments that need to actually dump cache after comments and new posts are posted, or posts are updated, all while traffic is hitting your uncached pages hard. Or if they are cached, they can end up over-cached and not reflect changes.
It does, unless you use variables (http://docs.blitz.io/variables) to spray a bunch of URL’s. You can also randomize query parameters for cache busting.
– @k0ws1k:twitter
Well, the question is, how was the test run here performed?
To a single URL, either, the newly created test post, or the index page, it didn’t really make a difference.
The parameters used are visible at the top of the blitz PDF’s at http://dl.dropbox.com/u/1503901/Blitz%20-%20final%20graphs.pdf , but I should have included them in the main body.
Pingback: Hello WordPress | Con[fusion]
Pingback: 10 Million hits a day with WordPress using a $15 server « -module(blog)
Hey I dont understand all this I have a blog too and I want to get 10 million hits/day. Please help me how can I get 10 million hits/day
Have something valuable to say.
ROFL!
Beware the smart quotes. Quite a few of the snippets in this post have smart quotes replacing the appropriate ” or ‘ quotes, e.g.:
GRANT ALL PRIVILEGES ON wordpress.* TO “wp_user”@”localhost”
If copy pasta isn’t working for you, that’s probably to blame. =)
Ewan,
Thanks for this interesting post! My project PressUpBox (https://github.com/mrdavidlaing/pressupbox) has very similar aspirations of configuring the optimal hosting environment for WordPress.
I’d like to do a performance comparison between your setup and mine.
Could you share a couple of further details so I could do this:
(1) Which theme & what content were you running during the test?
(2) Which pages were you hitting from the blitz.io during the test?
(3) Which processes were consuming memory, and how much did they each consume?
Thanks!
David
Theme was the standard Twenty Eleven theme.
Only hitting the single new post created, which isn’t the most accurate test, but then the vast majority of the people who visited this site only came to this page 🙂
Varnish was the main memory consumer, sorry I’ve not got accurate figures but if I had more memory for the test, I would have increased the Varnish memory allocation
Thanks for posting, this was helpful and inspired me to take a look at my wordpress install to see what can be improved.
Pingback: Dogfood « McGowanFam
Whoooaaa… amazing results when testing!! 🙂 Thank you for these tips!
Hey Ewan
I’ve being doing a load of testing recently on similar lines. Might I suggest you look at percona or mariadb which are faster drop in replacenents for MySQL.
Also I’d try php-xcache which would likely remove the need for the cache plugin and apc
Thanks Neil – I’d normally use something like Maria myself yes, it’s definitely better, but for this how-to/tutorial thing, I wanted to keep it as close to a “stock” Linux distribution as possible.
Pingback: New website | Gravity On Mars
Question: this page is loading in approximately two seconds. Did you switch to a configuration other than the one you detailed above?
No, but something odd is happening, I think an external javascript is hanging somewhere occassionally. Most of the time the page is loading instantly, but sometimes it just stops after the initial HTML render.
Try disabling DISQUS and see if there’s a change.
There are a lot of exceptions being generated. Probably you are using cloudflare which is not the best thing to use. I used that crap service from cloudflare as option is given in WordPress total cache. My recommendation is to get rid of cloudflare if you don;t want database not found error or unwanted captcha to be shown to genuine users,
WordPress in the past!Why generate a page each time a user comes, if it does not change?Much easier to generate a static site and carry generate each time changes are made. And the server is needed is the cheapest.
That’s Movable Type you’re talking about: http://www.movabletype.org/
No, not that! There is still jekyll, nanoc, and much more, much more of the best engines.
Yes, that.
What you’re talking about is exactly what a full page cache does 🙂
Caching pages – this is not static. Since each time, in any case verified page has changed or not.Thus, statics, which I create with the help of jekyll is orders of magnitude better than wordpress, which is needed for a specialized server.
See how it work my site www.juev.ru
Denis. The use of Varnish is for the problem you speak of. Varnish is pretty good at this and very much recommended as a Reverse Proxy. For example: If you set a TTL on the page to cache, it will never hit the WordPress software or its cache.
Varnish will say something like, “Oh your request looks like you want to see the post about baby polar bears, cool let me check if I have that cached. Oh I dont, let me go get it for you from WordPress. Here you go” Now any subsequent request for the page by anyone will be fetched by Varnish and *not* WordPress.
I know that this varnish, and I used it when I was on Wordpress.
However, you do not understand simple things you are doing more complicated systems. While you can just immediately get a static site with all the features of Wordpress.
Pingback: 15 Dolarlık Sunucuyla Günde 10 Milyon Hit | WordPress Notları
Great post. Glad to see more people proselytizing more ways to get performance than the usual “OK you got Apache installed and your site’s getting slow? Great, install W3 Total Cache or WP Supercache; OK getting slow again? Install memcached; getting slow yet again? Flip these knobs in W3TC; the end”.
A few thoughts and a question:
– If you’re using any kind of caching layer *in front* of WordPress you’ll want to exclude the WordPress admin pages from being cached. It can be confusing when you get a cached list of posts or cached edit post page, for example. – Similarly, you’ll probably want to make sure that users with the WordPress authentication cookie are excluded from cache, and you’ll need to make sure you remove anything that requires server-side logic from your theme and user-facing plugins, because your users will not be hitting the PHP backend of your server. If an authenticated user primes your cache, every other visitor will see their personalized content (such as the WordPress toolbar, their name in the comment form, etc). – You say “why” in some places, but not in others, and it would be very helpful to know the “why” for every instruction. e.g., why have fpm listen to a socket connection instead of a port connection, why use the /%post_id%/%postname% permalink structure, why am I supposed to enable object cache and database cache in W3 Total Cache — what are these things doing? The “why” will help someone make a better decision if/when something goes wrong, like unexpected things getting cached or if some other service using port 8080, and at least help point them in the right direction when they start looking for help on these things. – Why Varnish instead of nginx’s caching? nginx’s own ability to cache things is just as powerful as Varnish’s and can give you more resources since you’re running 1 less application in your stack, and uses files for caching so it’s less memory-intensive than Varnish. It also helps because users unfamiliar with using things like php-fpm and nginx will have 1 less application to learn/master. (Honest question here!)
The only thing that I still miss after all this todos – is the ftp access to the server. Could some of you step by step describe the process?!
Instead of using FTP, you should use SSH and it’s sister service, SFTP. That was all the data you transfer is encrypted – normal FTP even sends your own password in plain text.
SSH is enabled on the server by default, and you can use SFTP with it right away – from Windows you can install something like Filezilla, or WinSCP to do the file transfers.
Using filezilla, be sure the set protocol to SFTP and logon type to Interactive and you are good to go
The statement “10 million hits per day using WordPress on a box costing less than $20 a month” is wrong, 70.38MB of data pr minute amounts to ~101GB pr. day. The cost of the instance itself is one thing, the cost of the bandwidth is another. Combined cost is at least $360 pr. month if you use the numbers posted above.
Hi Egill, you’re entirely right, but that wasn’t really meant to be the point of the post.
Of course you’re going to have to pay for extra bandwidth, and if you actually had 10 million readers per day, then using a single EC2 micro instance would be more than a little silly, but the point of the post was that by optimising your installation, you can radically change the performance of your server.In this case, the “out of the box” figures were 900k hits per day with 30% error rates, and the improved results were over 10 million hits per day, all on a box with 512MB of RAM and limited CPU (but excellent network access).There’s all sorts of other things you need to pay for in a high volume site (backups, a content delivery network for static files, and a decent high availability setup being 3 obvious things), but there’ alsos huge amounts of improvement you can achieve over what most people consider to be “standard” wordpress install of Apache, and PHP with FCGI, that the WordPress installation docs talk about.
Pingback: 10 Million hits a day with Wordpress using a $15 server | Ewan’s Blog on IT and stuff like it | D3sordr3
Pingback: Bokmärken för April 2nd från 21:26 till 21:26 « Coverage
Pingback: The Ultimate Wordpress Setup
Pingback: Nginx, Varnish, PHP5-FPM, and Wordpress | WiseBison
Be sure and post if you try this again with removing varnish and expand nginx (and include setup) as mentioned in the comments. thx. bookmarking to try this this weekend.
I’ve had a quick look at it, it’s definitely fast but it’s not very friendly when it comes to applications like wordpress, drupal, etc. Unlike varnish, the nginx cache doesn’t automatically exclude things with cookies, so a logged in admin can mess up your cache, etc.
I need to do more reading around nginx’s implementation
If I were to run multiple domains off of the same nginx install this way, would the micro’s memory still be sufficient without needing to further distribute memory between each virtual site?
It should be fine really, especially if you can use one single php-fpm installation – just point each nginx
server {
server_name server1.com;
}
section to the same PHP section using:
fastcgi_pass unix:/dev/shm/php-fpm-www.sock;
nginx is incredibly effective for memory usage, so the only question is whether varnish would need a bigger cache, and that all depends on your traffic volumes and the number of different articles being read.
One thing I would say – you probably shouldn’t use an AWS micro instance for a machine running several production websites, they’re not the most reliable in the world.
Thanks. I mainly want to run a new, main blog, that isn’t established yet, but I have a couple of other domains I own and just want a placeholder up or host the minimal traffic it gets. 10 million hits per day is overkill, but it’s good to know I have room to grow.
You’re going to have to clarify your ‘they’re not the most reliable in the world’ statement though. Isn’t the whole point that I’ll have a stable WordPress install that can withstand traffic spikes? Should I use Linode instead of I want reliability?
EC2 instances with EBS attached can run for months without issue, or have a problem every week, it seems entirely random based on which physical host the instance happens to be deployed on (something you can’t control).
I’ve found Linode and Rackspace cloud to both be a bit more focussed on supporting people with a couple of VM’s rented from them than AWS, who don’t really support you until you get into high volumes of usage (this blog runs on a Linode 512MB VM and ran fine throughout the weekend with 30k+ readers).
Perfect. I’ll pay the extra $5/month and get another Linode instance then (I already use them for Rails anyway).
Running 3-4 virtual servers in nginx seems to push the CPU load generated by varnish up to 400% after about a day. None of the sites get any significant traffic (dozens to a few hundred page requests a day, not counting search engines), and if I comment out all but 1 or 2 of them, that doesn’t seem to happen.
I’ve got a 512 meg Linode VPS, and top says I’ve got about 30 megs free. Any idea what might be going on here?
Others have mentioned varnish may not actually be of much help here (despite, in your example, there’s obviously performance improvements. The tests in this article e.g. point out that it’s not very useful:
http://www.go2linux.org/linux/2011/04/nginx-varnish-compared-nginx-941
One slight difference form your setup is that I use a different permalink structure for my articles, but looking at the nginx and varnish config files, it doesn’t look like that matters.
The nginx CPU issue seems very odd, I can’t tell you what’s happening from the description alone but it’d be worth looking at enabling newrelic or another performance tracing system to see what’s going on.
For the varnish vs nginx caching performance, I did a little testing after some other people mentioned it, and you definitely can make nginx run as fast as varnish + nginx, but varnish handles things like cookies in a much more sensible (to me) way.
If you had an entirely static site, nginx with caching enabled would be fine, but for now I think I’ll stick to nginx + varnish
My tipps on the topic ( the results are quite the same )
Install CW Image Optimizer. This util needs Littleutils programs and exec rutins enabled for PHP; it optimizes uploaded images files on the fly. It also has a very good install guide. Google PageSpeed test shows much better values with this 🙂
In my opinion, don’t use database cache with WordPress. None of the plugins worked fully correctly for me. Tune MySQL instead. For example, add query cache, much better, than some PHP implementations.
APC Object Cache worked better for me than W3TC object cache.
I’ve made a cache plugin, which can be used together with nginx’s memcache pass: generating the page from PHP and served from nginx can be really speedy. Even if you set up 2-3 seconds you can save yourself from 100-200 req/sec rushes on a VPS like this.
Pingback: How I built this blog < NixFoo
Pingback: Varnish Running VCC-compiler failed on purge - Admins Goodies
I just setup a similar site and got similar (9,000,000) results from the following:
Apache + XCache + W3TC – more important for me than the ability to serve 9,000,000 pageloads a day (not likely on a blog) is the pageload time. I’d rather have 1,000 users who are happily loading content quickly!
The best performance increase comes (in my experience) with Apache by locking out .htaccess files (allowoverride None) and directly putting the WP permalinks code and W3TC code into httpd.conf (restart afterwards 😉 ). This really makes render times faster under heavy load.
Whatever you do, on these micro instances, things get tricky at the 250+ users per second level, but then, I would imagine, you’d probably be able to afford instances designed to handle this stuff!
You’re absolutely right about the .htaccess files – they’re a killer for performance on lots of sites, putting stuff into the httpd.conf file is much better, and also on page load times. If the site takes a minute to load (or even 10 seconds), that’s far too long for some people.
I’ve not played around with Apache and XCache combined, I’ll have to give it a try sometime.
Pingback: 10 Million hits a day with Wordpress using a $15 server | Ewan’s Blog on IT and stuff like it - Dougal Campbell's geek ramblings
Hey there, great post! I’m following it very well. I’m wondering… would you recommend one site per EC2 instance with this configuration or could you squeeze a couple onto the same server? Also, I’m having trouble setting up http://FTP... guess that’s just me being noob.
You can certainly run multiple sites on this same setup, I do on this server.
In the /etc/nginx/conf.d/default.conf file, simply copy the whole server { … } section, renaming the hostnames to be the second site, and install wordpress as normal. You can use the same varnish and php systems without any additional changes.
Instead of FTP, use SFTP – it’s an awful lot more secure, and handily SFTP is enabled by default as part of SSH. Use something like WinSCP if you’re on a windows server or Cyberduck on OS X to connect.
Thanks for the reply! I’m wondering, does WordPress allow you to still do automatic updates with the SFTP configuration you just mentioned?
If you follow these instructions, WordPress can update itself without prompting for any FTP or SFTP credentials, as the nginx user owns the wordpress PHP files.
Whether that’s secure enough is a question for a group of people in a pub – some people don’t like it, but I prefer the fact that people are more likely to apply updates if it’s seamless.
Thanks! I must have missed a step because when I go to install W3TC I get a prompt. I’m going to terminate my current instance and try again. Practice makes perfect…
One last question: I notice you install WP to /var/www/ in this tutorial. If I wanted to run multiple domains, an keep that directory nicely organized… could I install each domain like: /var/www/www.domain1.com and /var/www/www.domain2.com or is there another way I’m missing. Thanks, appreciate your responsiveness
That’s exactly what I do yes, though I leave off the www. at the front so it’s more immediately visible when you run a directory listing.
The most likely steps you’ve missed are:
chown -R nginx:nginx /var/wwwor:chown nginx:nginx /var/www/wp-config.php
Great article! It really works, thank you!
Pingback: 20.5 Million hits a day using an Amazon Micro server ($15 a month) and Node.js « The Wandering CIO
I went thought this article again and attempted to do it completely but with multiple domains under /var/www/domain1.com. When I get done with the WP install part and try to hit the domain I’m receiving “Welcome to Nginx” page and not sure how this is happening. Any help is appreciated.
It’s hard to know exactly without seeing all your system, but it might be that in the default.conf file, you need to change
https://github.com/ewanleith/Wordpress-Server-Configuration-Files/blob/master/default.conf
root /var/www/;
to
root /var/www/domain1.com/;
in the section
server {
## Your website name goes here. server_name domainname.com http://www.domainname.com;
That’s what I thought too… I made sure that the information in the default.conf has the proper domain information in it and it’s still showing up with the Nginx splash screen. Not sure if this is related, but when I try to restart nginx I receive “nginx: [emerg] unknown directive “port_in_redirect” in /etc/nginx/nginx.conf:1″ but that I’m assuming is because Varnish has not yet been setup.
Any other ideas of how I can get this going for multiple domains? Thanks for your responsiveness.
Until you fix the unknown directive message, you won’t be restarting nginx to make the other changes effective.
I’m not sure why you’re getting it, the /etc/nginx/nginx.conf file doesn’t need to change from the one for a single server.
You’ve put the line:
port_in_redirect off;
Inside the http { } brackets, right, and with a semi-colon at the end? You could delete that line for now, it’s not needed until varnish is running anyway.
Otherwise, I’m not sure what’s happening on your server, sorry.
Also, I copied the entire server { … } information so now it has 2 server { … } blocks of code. I’m assuming that’s proper.
Pingback: Hello, WordPress | Will Baumann's Blog
Pingback: How to Install Wordpress on AWS EC2 | The Art of Code
Pingback: nginx, wordpress & arm server Part 1 « Tom Gall
Pingback: The Growth Toolbox - Part 1 - Startups = Growth
Pingback: VPS dla bloga? Testy DigitalOcean - techfreak.pl
Pingback: WordPress as a document management system–Post Forking and Edit Flow Plugins | Anuj Varma, Technical Architect
Pingback: Простая настройка iptables для сайта | С миру по биту