I came across a VMware problem where several VM's were in a warning state with the following alert:
In English: "Virtual machine requires consolidation".
A right-click on the "VM -> Snapshot -> Consolidate" failed with the following error:
In English, the error message is "unable to access file
In the vmware.log file of this VM this could be followed and the virtual disk clearly failed to lock:
2015-04-02T07:01:00.608Z| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/mydatastoreuuid/myvm/myvm-000009.vmdk' : Failed to lock the file (16392) 6873
2015-04-02T07:01:00.753Z| vcpu-0| I120: Vix: [173318 vigorCommands.c:548]: VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)
This happened on several VM's on several physical hosts. One thing was in common though: The vCenter server was recently updated and therefore there was a communication loss between vCenter server and the physical ESXi hosts.
The VMware KB entry 2003638 contains an interesting note:
The snapshot commit fails due to locked files. For details on how to release the lock on locked files, see:
Investigating virtual machine file locks on ESXi/ESX (10051)
Unable to delete the virtual machine snapshot due to locked files (2017072)
There is a temporary loss of communication between the vCenter Server and the ESXi/ESX host during snapshot commit. The most common cause of this is an environmental issue which can have a knock on affect on the ESXi/ESX host management agent hostd. For troubleshooting guidance and some possible causes of such an outage, see Troubleshooting vmware-hostd service if it fails or stops responding on an ESX/ESXi host (1002849).
Actually both hints (the lock and loss of communication) combined present the reason and the solution.
There was a loss of communication between the vCenter Server and the ESXi hosts (due to the vCenter update). Yet it is also very likely, that the backup software tried to create a snapshot during this communication loss.
The solution now is to force a re-connection of the ESXi server to the vCenter server. This can be done by restarting the hostd service:
After the hostd restart, wait 1 to 2 minutes and then launch "Consolidate" again.
This has worked fine on all VM's and solved the warnings.
No comments yet.
AWS Android Ansible Apache Apple Atlassian Automation BSD Backup Bash Bluecoat CMS Chef Cloud Consul Container Containers CouchDB DB DNS Database Databases Docker ELK ElasticSearch Elasticsearch Filebeat FreeBSD GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java Kibana Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Rancher SSL Security Shell SmartOS Solaris Surveillance SystemD TLS Tomcat Ubuntu Unix VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder