These days I'm testing Rancher as a potential candidate for a new Docker infrastructure. It's appealing so far: Rancher does have a nice and intuitive user interface and more importantly a nice API to automatically trigger container creation (for example through Travis).
During a failover test, I rebooted one of the Rancher hosts and when it came back up, the connectivity to Rancher was lost. Why? Because I forgot to add the separate file system for /var/lib/docker, which I prepared as a logical volume, into /etc/fstab - therefore all previous docker data was gone and of course also the rancher-agent container.
Unfortunately I didn't see the error as fast and I just decided to simply remove the host in Rancher and re-add it manually. Of course when I fixed the file system mount problem and rebooted, Rancher would not connect anymore, because meanwhile there is a new rancher-agent with a new ID installed.
To force a reset or cleanup of the Rancher host, one can do the following:
1. Deactivate the affected host in Rancher, then remove the host
2. Stop Docker service
service docker stop
3. Remove Docker and Rancher data:
rm -rf /var/lib/docker/*
rm -rf /var/lib/rancher/*
4. Start Docker service
service docker start
5. Add the host in Rancher
Update August 17th 2018
The above commands apply to a Rancher 1.x environment. In Rancher 2.x more directories must be cleaned up:
1. Deactivate the affected host in Rancher, then remove the host
2. Stop Docker service
service docker stop
3. Remove Docker, Rancher, RKE and Kubernetes related data:
rm -rf /var/lib/docker/*
rm -rf /var/lib/rancher/*
rm -rf /var/lib/etcd
rm -rf /etc/kubernetes
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/cni
rm -rf /var/run/calico
4. Start Docker service
service docker start
5. Add the host into a cluster in Rancher UI
No comments yet.
Personal Internet VMware PHP Linux Shell Bluecoat Proxy Windows Hardware Virtualization Nagios MySQL DB Monitoring Mail Android Network Wyse Hacks Tomcat Postgres Apple Mac Backup BSD ZFS Solaris SmartOS Unix Multimedia Perl Database MongoDB CMS OTRS FreeBSD Wordpress LXC Nginx Proxmox DNS Graphics GlusterFS Security Chef HAProxy Icinga Ansible HTML MariaDB Containers Rancher Docker AWS ELK Kibana Logstash Filebeat Varnish PGSQL PostgreSQL ElasticSearch CouchDB Bash Macintosh Container Minio Grafana InfluxDB Databases NFS OSSEC SystemD Java Zoneminder Surveillance Elasticsearch SSL TLS Icingaweb2 Cloud Wireless Kubernetes Ubuntu