Mount a GlusterFS volume in a LXC container

Written by - 5 comments

Published on September 1st 2014 - Listed in Linux LXC GlusterFS

Unfortunately it is not as easy to mount a GlusterFS volume in a LXC container as it is with another device, for example an additional Logical Volume. 

I first tried it with the LXC's fstab file:

cat /var/lib/lxc/lxcname/fstab
localhost:/vol1 mnt glusterfs defaults,_netdev 0 2

This should (in theory) mount the GlusterFS volume "vol1" from localhost into the LXC container with mountpoint /mnt. Yes, the missing slash is correct given the relative path to the LXC container.

But unfortunately, this didn't work as a start of the container in debug mode showed:

lxc-start -n lxcname -o /var/lib/lxc/lxcname/stdout.log -l debug

cat /var/lib/lxc/lxcname/stdout.log
lxc-start 1409577107.058 ERROR    lxc_conf - No such device - failed to mount 'localhost:/vol1' on '/usr/lib/x86_64-linux-gnu/lxc/mnt'
lxc-start 1409577107.058 ERROR    lxc_conf - failed to setup the mounts for 'lxcname'
lxc-start 1409577107.058 ERROR    lxc_start - failed to setup the container
lxc-start 1409577107.058 ERROR    lxc_sync - invalid sequence number 1. expected 2

As a second attempt, I tried it within the LXC container (as with a normal Linux host) in /etc/fstab:

cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM /mnt glusterfs defaults,_netdev 0 0

Where is the IP address of the physical host of this LXC container.
Before rebooting the container, I tried to mount the gluster volume manually:

mount.glusterfs /mnt
Mount failed. Please check the log file for more details.

Ah crap! What now? I checked the glusterfs mount log:

cat /var/log/glusterfs/mnt.log
I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.2 (/usr/sbin/glusterfs --volfile-id=/vol1 --volfile-server= /mnt)
E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory)
E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again

Indeed, the special character device /dev/fuse is missing in the container while on the physical host it exists.
At first I thought this is a device permission issue which needs to be solved in the container's config file but the relevant config for /dev/fuse is already there by default:

cat /usr/share/lxc/config/ubuntu.common.conf | grep -A 1 "## fuse"
## fuse
lxc.cgroup.devices.allow = c 10:229 rwm

Then I stumbled across this Github issue: where St├ęphane Graber, one of the LXC's main developers, answered this:

Some modules will also require the creation of device nodes in the container which you'll need to do by hand or through init scripts.

So to solve this, I created /dev/fuse manually within the container:

mknod /dev/fuse c 10 229 

And then tried the manual mount again:

mount.glusterfs /mnt

No error this time. Verification with df:

df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/lxc/lxcname          20G  830M   18G   5% /
none                     4.0K     0  4.0K   0% /sys/fs/cgroup
none                      13G   60K   13G   1% /run
none                     5.0M     0  5.0M   0% /run/lock
none                      63G     0   63G   0% /run/shm
none                     100M     0  100M   0% /run/user         99G   60M   99G   1% /mnt

To simplify such things, the /dev/fuse can of course already be created during "lxc-create" by modifying the relevant lxc template. Saves you a headache later.

Add a comment

Show form to leave a comment

Comments (newest first)

Dino from wrote on Jan 12th, 2017:

I believe I ran into this issue as well with /dev/fuse

ck from Wil, Switzerland wrote on Dec 14th, 2015:

Hi Mark. In my case it worked. Once created within the container, the device node staid. Verify if the permissions (lxc.cgroup.devices.allow) are correct so the container is allowed to access the devices. I haven't tried glusterfs mounts yet with newer LXC releases.

Mark from wrote on Dec 13th, 2015:

But how do we actually get the 'mknod /dev/fuse c 10 229' to be persistent across reboots and also the glusterfs mounts? I haven't had any luck with either.

Ovidiu from San Francisco Bay Area wrote on May 22nd, 2015:

Sadly this doesn't seem to be working with LXD 0.9, where containers run as unprivileged users.

Adam from Boone, NC wrote on Jan 2nd, 2015:

Thanks for posting this quick guide. I just ran into this issue of mounting a GlusterFS inside a LXC container and would have spent a lot of time tracking down the /dev/fuse issue.

Thanks again,