Gluster is a fantastic open-source clustering filesystem, allowing you to convert low-cost Linux servers into a single highly available storage array.
The project has recently launched the “Gluster Storage Platform”, which integrates the Gluster filesystem with an operating system and management layer, but if you want to add Gluster functionality to your existing servers without turning them into dedicated storage appliances, the documentation is a bit lacking.
In an attempt to help anyone else out there to get Gluster up and running replicating a directory between 2 servers in Gluster’s “RAID 1” mode.
First of all, download the latest version of Gluster 3 from their FTP site, I downloaded 3.0.0-1, you’ll need the following files assuming you’re running CentOS:
glusterfs-client-3.0.0-1.x86_64.rpm
glusterfs-common-3.0.0-1.x86_64.rpm
glusterfs-server-3.0.0-1.x86_64.rpm
Once you’ve downloaded the 3 files to somewhere on your first node, run:
yum install libibverbs
rpm -ivh glusterfs-client-3.0.0-1.x86_64.rpm glusterfs-common-3.0.0-1.x86_64.rpm glusterfs-server-3.0.0-1.x86_64.rpm
To install the Gluster software itself. Then copy the RPM files to your second node, and repeat the rpm installation.
You’ll need to decide a directory on each server to act as the datastore, so either pick an existing directory or more likely create a new one – In this case I’ve used “/home/export”. If it doesn’t already exist, run
mkdir /home/export
Assuming you’re using 2 nodes, next run this command on the first node to produce the Gluster configuration files, replacing the words node1ip and node2ip with the IP addresses or hostnames of the 2 nodes, and /home/export with your directory.
glusterfs-volgen —name store1 —raid 1 node1ip:/home/export node2ip:/home/export
This will create 4 files
booster.fstab
store1-tcp.vol
node1ip-store1-export.vol
node2ip-store1-export.vol
Of these, booster.fstab is used for auto-mounting filesystems after reboots, so isn’t needed yet. Copy the store1-tcp.vol and node2ip-store1-export.vol files to the second node.
On the first node, run
cp node1ip-store1-export.vol /etc/glusterfs/glusterfsd.vol
cp store1-tcp.vol /etc/glusterfs/glusterfs.vol
On the second node, run
cp node2ip-store1-export.vol /etc/glusterfs/glusterfsd.vol
cp store1-tcp.vol /etc/glusterfs/glusterfs.vol
At this point, you should be ready to start the gluster services on both nodes, and mount the filesystems.
You need somewhere on the node to mount the replicated filesystem, In this case we’re using “/mnt/export”
On each node, run
service glusterfsd start
mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/export/
You should now have a working Gluster replication service between the 2 nodes, you can test this by running on the first node
echo “Gluster is working” > /mnt/export/fileA
and on the second node run
cat /mnt/export/fileA
Assuming everything is working ok, you’ll see a message of “Gluster if working” on your screen.
If you don’t get that, then take a look in the /var/log/glusterfs/ directory on both nodes to see what’s happening.
One thing I’ve noticed is that gluster’s output logs in /var/log/glusterfs often start with a – at the front, confusing a lot of Unix command line tools – if you refer to them using their full path including /var/log/glusterfs, you’ll have an easier time manipulating them.
Hopefully this will help people out there with the slightly confusing but comprehensive Gluster documentation where I recommend you go for any more in-depth configuration help.