Notes on Deploying a Two-Node HA Cluster and GlusterFS Distributed File System on Proxmox VE
Deploying a Two-Node HA Cluster and GlusterFS Distributed File System on Proxmox VE
Make sure both pve nodes run the same version
Two proxmox ve nodes, enabling high availability with automatic vm and lxc migration
Edit the hosts file
In /etc/hosts on both pve machines, add the following host entries:
|
|
Install glusterfs
The following operations are performed on both machines, referred to here as pve1 and pve2
|
|
Make sure the gluster versions are the same
|
|
If they are the same, upgrade them. One of my machines was 9.2, and even after adding the source it wouldn’t upgrade directly to 10.3, so you need to do this:
|
|
It will list all available versions; then install the corresponding one
|
|
On pve1, edit nano /etc/glusterfs/glusterd.vol, and under option transport.socket.listen-port 24007 add:
|
|
On pve2, edit nano /etc/glusterfs/glusterd.vol, and under option transport.socket.listen-port 24007 add:
|
|
Start the service
|
|
Important: on pve2 you need to run the command to join the cluster
|
|
It’s OK once it shows: peer probe: success
Add a volume
Here it is created directly on zfs; make sure both pve nodes use the same path to create the volume, which is /tank/gfs here
Note: if it’s zfs, don’t use the mount root itself, but use a directory. In the example below, /tank/gfs is a zfs mount, and a subdirectory s is created under it to serve as the volume
|
|
Check the status
|
|
Add the mount
Do this on both pve nodes
|
|
Edit pve1’s /etc/fstab and add
|
|
Edit pve2’s /etc/fstab and add
|
|
Reboot both pve nodes to let /mnt mount
Mount without rebooting both pve nodes
|
|
Solving the split-brain problem
With two nodes, gluster can run into a split-brain problem — the two nodes have equal votes and neither will listen to the other. The solution is as follows:
|
|
Setting up the two-node pve cluster
Create on the first, join on the second — nothing much to say
Create a shared directory
In Storage under DataCenter, click Add, fill in /vms for Directory, and check share
HA setup
Edit /etc/pve/corosync.conf
Add to quorum so it becomes like this:
|
|
Here, expected_votes indicates the desired number of nodes, and two_node: 1 means there are only two nodes. There is also a wait_for_all: 0, NOTES: enabling two_node: 1 automatically enables wait_for_all. It is still possible to override wait_for_all by explicitly setting it to 0. If more than 2 nodes join the cluster, the two_node option is automatically disabled.