These days I'm testing Rancher as a potential candidate for a new Docker infrastructure. It's appealing so far: Rancher does have a nice and intuitive user interface and more importantly a nice API to automatically trigger container creation (for example through Travis).

During a failover test, I rebooted one of the Rancher hosts and when it came back up, the connectivity to Rancher was lost. Why? Because I forgot to add the separate file system for /var/lib/docker, which I prepared as a logical volume, into /etc/fstab - therefore all previous docker data was gone and of course also the rancher-agent container.

Unfortunately I didn't see the error as fast and I just decided to simply remove the host in Rancher and re-add it manually. Of course when I fixed the file system mount problem and rebooted, Rancher would not connect anymore, because meanwhile there is a new rancher-agent with a new ID installed.

To force a reset or cleanup of the Rancher host, one can do the following:

1. Deactivate the affected host in Rancher, then remove the host

2. Stop Docker service

service docker stop

3. Remove Docker and Rancher data:

rm -rf /var/lib/docker/*
rm -rf /var/lib/rancher/*

4. Start Docker service

service docker start

5. Add the host in Rancher

Update August 17th 2018

The above commands apply to a Rancher 1.x environment. In Rancher 2.x more directories must be cleaned up:

1. Deactivate the affected host in Rancher, then remove the host

2. Stop Docker service 

service docker stop

3. Remove Docker, Rancher, RKE and Kubernetes related data:

rm -rf /var/lib/docker/*
rm -rf /var/lib/rancher/*
rm -rf /var/lib/etcd
rm -rf /etc/kubernetes
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/cni
rm -rf /var/run/calico

4. Start Docker service

service docker start

5. Add the host into a cluster in Rancher UI