I’ve previously blogged about my belief that ARM servers are a big part of the future, and that server manufacturers are missing a trick by not testing the market with small, cheap, ARM servers, but I wondered if maybe they knew something I didn’t.
To find out what I was missing, I bought the closest thing I could find to a generic ARM server, the Beaglebone, an ARM development board from Texas Instruments, with a TI AM3358 ARM Cortex-A8-based microprocessor, 256MB of RAM, 100Mb ethernet, a couple of USB ports, a micro-SD card slot and not much else.
I’ve had the beaglebone for a few months now, and done various things like installed Ubuntu on it, put it in a Beaglebone custom case so it’s not a bare board anymore, run nodejs services, a VPN to get home network access from my phone and laptop, and generally used it as if it were a normal server, without any real issues.
So, now it’s time to take it to the next level – I’ve paid for the beaglebone to go into a colocation rack in Telehouse North, with a friendly colocation company called Jump Networks who were happy to help out with the experiment, and who only charge for £50 + VAT for installation and a very low monthly cost for hosting the equipment, as little as £5 per month – perfect for an ARM server.
And now, my tiny ARM server is live, running on the Internet, doing real stuff for my work, while we learn the limitations and issues of running an ARM server in the wild.
So far, I’ve found out the following things:
Of these, only the third one has been a real issue for far – I’ve already had one SD card go bad while running the bone at home. If it happens again I’ll need to send a new pre-configured SD card to insert into the slot. The lack of a serial console hasn’t been a problem yet, but I expect it to be at the most awkward moment possible…
All 3 problems are solvable, for a price, the Beagleboard XM for example has a serial console, and 512MB of RAM, but costs £125 as it comes with lots of other features a server doesn’t need, like DVI-D and S-Video ports, along with a slightly faster 1GHz CPU. While the price isn’t “high”, it’s twice the price of the Beaglebone.
So for now, the Beaglebone will run day to day, doing real world things, and perhaps I’ll find out why server manufacturers aren’t making these things, and why hosting companies aren’t buying them in bulk yet. In the mean time, if you’ve got any questions about the Beaglebone, let me know and I’ll answer them the best I can.
Update – I’ve now configured the Beaglebone to run WordPress with nginx caching and done some pretty successful load testing on it.
Raspberry Pi, here we come. Not a lot of RAM on that too either 🙁
Beowulf cluster would be a good idea (evil laugh).
I know you’re half joking, but a custom case designed to hold 50 of these in a 2U space would be as dense as any virtualised system
though i guess not arm, you might like the upcoming apc via board, also what about the raspberry pi in similar circumstance -> apc link
http://apc.io/
Yeah I think the APC board looks really nice, it’s great to see people are showing in an interest in them. I do think VIA should’ve put it in a case though, bare boards are nice to look at, but not so nice for damage from static electricity..
I’m looking forward to your conclusion !
Nice experiments man
Microsoft is concentrating on an ARM build for Windows, presumably for tablets. An ARM build for Windows server might be mandatory if commodity servers and OSes flood the market. Exciting innovations.
Re issue #3
Did you try upgrading to a class 4 or higher SD card? From what I know, the default speed for these devices is “class 2”. I hit the same problem with my smartphone and (mp3s skipping and such) and these were gone with a class 6 inserted.
I did yes, I had a Samsung 8GB Class 10 Micro SDHC which failed horribly shortly before I shipped it to the colocation, and replaced it with a class 6 card, which is currently running ok.
The disk throughput is decent, but lots of little writes can cause it to crawl.
IIRC The class 6 and up cards are optimised for streaming writes: they’re sold for use in cameras and camcorders. If your use case doesn’t include large streaming writes then there’s no point buying a card with a high class number.
I’ve seen some reports that class 2/4 cards actually perform better on random read/write benchmarks but I’ve never put this to the test myself.
I think you should consider a SLC card, for a longer lifespan (not much fun to change card in a colocated server) and better performance.
For instance: http://www.newit.co.uk/shop/proddetail.php?prod=Class_10_Integral
I looked for something exactly like this when I first bought the board, but couldn’t find one.
Thanks!
Interesting post, I’ve been thinking about this as well. Instead of upgrading to a newer laptop, or workstation, simply adding in a new rack (so to speak). The trick, I feel, is creating a home station that is basically a mini-service-oriented-architecture. Additionally, a use might be to duplicate the “cloud” locally, since running it on a single machine doesn’t quite match the cloud deployment environment. For instance, instead of using Dokuen (https://github.com/peterkeen/dokuen) you might have a little rack of mini arm computers.
Are you hosting this WP blog on the ARM device (as part of the normal server usage)? It might be interesting to see how it deals with the HN crowd 🙂
Great post and good on you for putting your theory into action.
No, I’ve not dared “expose” it to unpredicatable users yet 🙂
It’s currently running a few services that I control, but I will try flipping the blog traffic over to it sometime soon and see what happens..
I run similar set ups using Sheeva plugs from Globalscale, and what I’ve found to be useful is to keep a bare root fs mounted read-only on an SD card (remounting rw for updates and to install software), and keeping /home and /var on a usb hard drive.
We used to build Flash disk based encryption devices as far back as 1999, when writing to a Flash disk was still something you had to think about very carefully, since they had a very limited write cycle lifetime.
We achieved very good performance by using picoBSD (comes with the FreeBSD source distribution), and running the entire system on a 32M ramdisk. (A lot like you would do when building a boot floppy).
We would then only mount the flash disk when configuration changes or firmware upgrades took place. This took the flash disk out of the loop almost entirely.
Sounds like something you could do with the SD card too.
Great experiment, I’m very excited to see the results.
SheevaPlug. Why did Marvell drop the ball?
My SheevaPlugs have been serving a 20K subscriber RSS feed 24/7 for 2.5 years
“servers really need Ethernet or Serial consoles”… Serial consoles… The SERIAL CONSOLES called, they want their dodgy 1960s tech back. Why do you hate society and progress, you Luddite?
Because, rather surprisingly, that’s what most colocation providers offer as alternative access to your server in case the network fails…
It might not be so surprising why serial out-of-band (or ethernet, usually doing ssh to serial console anyway) is fantastic in the following situations 🙂 :
1) Networking failed, perhaps you (or an initscript failing to do a networking restart fully) cause you to lose network access, the only way to fix it is over something other than your downed network port – serial console saves the day. (some servers do ethernet console shared with a normal networking port, but this is no help if you’ve downed the physical link to the switch – hence why serial or dedicated ethernet oob is valuable)
2) There’s a kernel panic. What happened? You just go straight to what you’ve captured from your serial console and you’ve got the whole thing there for copy and paste to submit bug reports on, or search with.
There’s also plenty of servers with IPKVM these days too, but this is less useful when you have a kernel panic as you can’t cut and paste the crashdump, and noone wants to transcribe a screenshot of register dumps into the plaintext it came from, so text based really wins here! Maybe when you can paste a screenshot of a panic into google the text based method will be less of a concern.
Of course IPKVM is rather handy if you’re one of the few running windows as your server os.
Those are all good reasons, but the surprise is that something like 30 years after serial ports first became common, there’s still nothing better!
Been looking into that a lot and the unfortunate truth is nobody is currently producing a cost-effective alternative to x86, that cortex a8 cpu is so weak it would take more than 2000 bucks of beaglebones to match a light bulldozer virtualization host that cost 500.
The big reason the server makers and INTEL don’t rush into ARM is that it’s going to slaughter the margins, as you can source anything from anyone and the ARM license is cheap.
In the end, most of the stuff will be ARM or another RISC that has much more open competition, but all the players will delay that as long as they can.
I build custom ARM hardware for a living. I promise, if you’re a company with high volume working at the component level rather than an individual buying off the shelf single-board-computers, ARM is much, much less expensive than x86.
You hit the nail on the head about low margins/commoditization, though. However, if the value proposition is there, it might be a rain maker for a smaller player that’s willing to accept low margins for revenue growth. Dell probably isn’t going to start selling ARM servers right away because the margins on x86 are too fat, but with decent funding Joe Blow Electrical Engineer and his buddy Larry the Layout Guy can put together a very inexpensive Cortex board no problem.
Annnd I’ve hit my buzzword limit for the day, folks.
nice experiment.
http://www.embeddedarm.com/ run their site on an arm board:
http://www.embeddedarm.com/about/resource.php?item=257
Nice board, I have one, but I have never tried to connect the SATA disk ports.
Phil
I’m guessing one reason mini-servers aren’t more popular in datacenters is that you’d have more hardware. Each box needs to be racked, have it’s own network cable, etc. Yoi also lose the flexibility thay comes with running on a VM, especially the ability to resize your.box as needed.
That’s certainly true on a small scale, but if a datacenter was hosting 100s or 1000s of these things, there’s nothing stopping them building custom chassis’ to hold them all, much like a blade center.
I’ve ordered a CuBox from Solid Run (99 EUR + shipping) and I’m happy so far: Ubuntu Lucid, 1 GB DDR3@800 RAM, Gigabit Ethernet, 2 x USB (I’m using an external HDD), more info here:
http://www.solid-run.com/products/cubox
You probably know, but SD cards are essentially miniature SSDs, except they’re worse for reliability under moderate to high write load because all of the wear leveling is going into the same chip. I don’t remember if the bone has a USB host or not, but if you could plug in an external HDD and use that for any write-intensive operations, you’ll have much better luck.
Also, no matter what the controller on the SD card does, you’re still limited by the write speed of the NAND. For higher density MLC devices, this can get quite slow. SSDs get around this by interleaving writes across multiple cells/chips, but again, SD cards are limited in this way.
The problem with running an external HDD is power consumption will be massively higher for the HDD than the entire beaglebone, which runs at under 0.5W 🙁
When running Ubuntu on an ARM board from SD Card I noticed that there is copious amounts of logging which effectively is killing the SD Card bit by bit. Have a look at /var/log/* to find out which files are seeing frequent updates and lessen the log levels for stuff you don’t care about. Might help extending the life of your cards…
How about this bad boy?
http://www.itworldcanada.com/blogs/android/2012/05/23/the-name-is-pc-android-pc/63680/
It may be a banal comment, but you really should try using RasPi!
The OMAP hacking community (in academia mostly) has substantially addressed all of your issues to a fairly acceptable standard. The Sandia Strongbox uses both serial and usbnet to supplement a rather anemic ethernet. Some of these wouldn’t be appropriate for your usage but on our Pandaboard cluster we get decent WattsUp meter measurements because of our node count, and we boot from USB using a variant of https://groups.google.com/group/pandaboard/browse_thread/thread/f73e5e6c7127baab?pli=1
Pingback: Sysadmin Sunday 81 « Server Density Blog