VMware bug causes worldwide chaos
VMware virtual machines on all hosts with the company's latest hypervisor, ESX 3.5 Update 2, in enterprise configurations have found that it will not power on after being turned off.
The hypervisor refuses to start when the date is 12 August, with customers around the world discovering the problem as midnight was passed in their time zones.
A flaw in the VMware licensing code is responsible, according to Martin Niemar, group manager of virtual infrastructure product marketing at VMware, the most popular player in virtualisation.
"We had an issue with 3.5 Update 2. It's actually a licensing problem," said Niemar. "Currently, what we know is that licensing prevents new virtual machines from powering up after shutdowns, and it prevents virtual motioning, moving a virtual machine from one host to another."
Niemar said that VMware did not have a patch, but was working on one as a "top priority".
"Customers should not stop virtual machines. Keep virtual machines going until we release a patch," said Niemar. "You can also move the clock backwards on the server."
Some organisations cannot turn server clocks back for legal or technical reasons. Niemar said that, if customers have to turn machines off, and cannot turn clocks back, there is currently no fix. Niemar could not commit to a timescale for a patch, nor could he comment on forum claims that the fix will first be available to customers as a complete re-installation from ISO or TAR images, with a patch for installed code coming later.
"We understand the bug," he said.
VMware first learnt of the issue when Asia-Pacific customers started to come online on 12 August.
Technical issues have been discussed on the VMware communities blog.
One poster stated VMware had told them that "VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media, including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade TAR and ZIP files by noon, PST, on 13 August. These will be available from the page: http://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2. The Update patch bundles will be released separately, later in the week."
At the time of writing, this was unable to be confirmed.