I came across a VMware problem where several VM's were in a warning state with the following alert:
In English: "Virtual machine requires consolidation".
A right-click on the "VM -> Snapshot -> Consolidate" failed with the following error:
In English, the error message is "unable to access file
In the vmware.log file of this VM this could be followed and the virtual disk clearly failed to lock:
2015-04-02T07:01:00.608Z| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/mydatastoreuuid/myvm/myvm-000009.vmdk' : Failed to lock the file (16392) 6873
2015-04-02T07:01:00.753Z| vcpu-0| I120: Vix: [173318 vigorCommands.c:548]: VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)
This happened on several VM's on several physical hosts. One thing was in common though: The vCenter server was recently updated and therefore there was a communication loss between vCenter server and the physical ESXi hosts.
The VMware KB entry 2003638 contains an interesting note:
The snapshot commit fails due to locked files. For details on how to release the lock on locked files, see:
Investigating virtual machine file locks on ESXi/ESX (10051)
Unable to delete the virtual machine snapshot due to locked files (2017072)
There is a temporary loss of communication between the vCenter Server and the ESXi/ESX host during snapshot commit. The most common cause of this is an environmental issue which can have a knock on affect on the ESXi/ESX host management agent hostd. For troubleshooting guidance and some possible causes of such an outage, see Troubleshooting vmware-hostd service if it fails or stops responding on an ESX/ESXi host (1002849).
Actually both hints (the lock and loss of communication) combined present the reason and the solution.
There was a loss of communication between the vCenter Server and the ESXi hosts (due to the vCenter update). Yet it is also very likely, that the backup software tried to create a snapshot during this communication loss.
The solution now is to force a re-connection of the ESXi server to the vCenter server. This can be done by restarting the hostd service:
After the hostd restart, wait 1 to 2 minutes and then launch "Consolidate" again.
This has worked fine on all VM's and solved the warnings.
No comments yet.
Personal Internet VMware PHP Linux Shell Bluecoat Proxy Windows Hardware Virtualization Nagios MySQL DB Monitoring Mail Android Network Wyse Hacks Tomcat Postgres Apple Mac Backup BSD ZFS Solaris SmartOS Unix Multimedia Perl Database MongoDB CMS OTRS FreeBSD Wordpress LXC Nginx Proxmox DNS Graphics GlusterFS Security Chef HAProxy Icinga Ansible HTML MariaDB Containers Rancher Docker AWS ELK Kibana Logstash Filebeat Varnish PGSQL PostgreSQL ElasticSearch CouchDB Bash Macintosh Container Minio Grafana InfluxDB Databases NFS OSSEC SystemD Java Zoneminder Surveillance Elasticsearch SSL TLS Icingaweb2 Cloud Wireless Kubernetes Ubuntu