/ Security

Meltdown and Spectre

2018 is certainly kicking off with an interesting set of security issues that affect all processors in use today, Intel, AMD, ARM and so on.

In short the three vulnerbilities are:

Meltdown - CVE-2017-5754
Spectre - CVE-2017-5753 and CVE-2017-5715

Azure and AWS have already patched their hosts. They were one of the first to respond mostly because exploiting meltdown allows access to other VM's. In a heavily shared environment like Azure and AWS it would mean that access would be granted to other customers machines. It's important to note that this will not provide protection to your VM's until you've installed the OS level patches. The host level patches just allow the OS level patches to work. when they are installed. This is the same for VMWare.

The CPU Impact

This is where things get a little contentious. Intel admit that yes, there is a CPU impact when this patch is installed. Intel themselves are coming out with patches but there has been no information forthcoming from them on if the CPU level patches mean that the OS level patches are still required. It's all a bit of a mess.

Epic Games posted an interesting article that showed how a cloud based server was impacting post patching: https://www.theverge.com/2018/1/6/16857878/meltdown-cpu-performance-issues-epic-games-fortnite

The problem that I have with this article is the sheer amount of unknowns.
We don't know what the load is on those servers - i.e. is it data modelling, is it database based, something else?
We don't know what processors those stats are based on. Older processors are known to have a harder time with the patches.
We also don't know the OS involved and the patch levels of that OS so until more information is out, I'd take this as something of an advsisory but then again, ideally all patches should be tested in a lab/UAT environment before production.

Intel have released some real world impact numbers in a PDF which is worth a read and can be found here: https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Blog-Benchmark-Table.pdf

The Patches

Hypervisor Level

VMWare:
VMware vSphere 6.5: apply patch ESXi650-201712101-SG (released on Dec, 19th 2017)
VMware vSphere 6.0: apply patch ESXi600-201711101-SG
VMware vSphere 5.5: apply patch ESXi550-201709101-SG (this patch has remediation against CVE-2017-5715 but not against CVE-2017-5753)

VMWare also have an advisory here: https://www.vmware.com/security/advisories/VMSA-2018-0004.html

OS Level
Windows Server, version 1709 (Server Core Installation) 4056892
Windows Server 2016 https://support.microsoft.com/en-gb/help/4056890/windows-10-update-kb4056890
Windows Server 2012 R2 https://support.microsoft.com/ur-pk/help/4056898/windows-81-update-kb4056898
Windows Server 2012 Not available
Windows Server 2008 R2 https://support.microsoft.com/en-us/help/4056897/windows-7-update-kb4056897
Windows Server 2008 Not available

On Windows platforms, a registry key needs to be set to turn the patch on:

To enable the fix

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

If this is a Hyper-V host and the firmware updates have been applied: fully shutdown all Virtual Machines (to enable the firmware related mitigation for VMs you have to have the firmware update applied on the host before the VM starts).

Restart the server for changes to take effect.

To disable this fix

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 3 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

Restart the server for the changes to take effect.

(There is no need to change MinVmVersionForCpuBasedMitigations.)

This key is supposed to be set by AV software so that it can say that it doesn't use the vulneribilties to do it's job. Software that does will break when this patch is deployed. I've never seen MS do this before but in this case, installing the patch doesn't mean that you're protected. That registry key must be set.

Testing Meltdown and Spectre

There is a Windows powershell test to see if a machine is vulnerable:

PowerShell Verification
Install the PowerShell module
PS > Install-Module SpeculationControl
Run the PowerShell module to validate protections are enabled
PS > Get-SpeculationControlSettings
https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution

Firmware Updates

Dell have released some firmware updates but only for the newer gen servers, updates for older servers are due out in February. http://www.dell.com/support/contents/us/en/19/article/product-support/self-support-knowledgebase/software-and-downloads/support-for-meltdown-and-spectre

Should I buy new hardware now or wait?

This is a question I've seen asked a few times and the answer is simply that it doesn't matter. Right now, neither Spectre nor Meltdown are being actively exploited (unless you consider bad AV software an exploit). There is far, far more risk from attack vectors such as email based ransomware. Of course, this may well change.

Updates

9th January - MS Pull patches due to problems https://www.computerworld.com/article/3246188/microsoft-windows/microsoft-yanks-buggy-windows-meltdown-spectre-patches-for-amd-computers.html

12th January - Kevin Beaumont has put up a google doc listing various AV software and how well they handle the registry key and if they are even compatible with the MS issued patch (thanks to Jason Dance for linking the document to me): https://docs.google.com/spreadsheets/d/184wcDt9I9TUNFFbsAVLpzAtckQxYiuirADzf3cL42FQ/edit#gid=0

15th January - VMWare pull patches https://www.reddit.com/r/sysadmin/comments/7qjnfx/vmware_pulled_spectre_patches_on_friday/

23rd January - Intel say "don't patch meltodwn" https://thehackernews.com/2018/01/intel-meltdown-spectre-patch.html

Handy Links

nixCraft have posted a document on how to update intel firmware from within Linux: https://www.cyberciti.biz/faq/install-update-intel-microcode-firmware-linux/

On a more worrying note, Dell and removed the firmware patch for Gen13 machines.
dell-13g

Finally, Intel have provided some real world testing figures: https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Blog-Benchmark-Table.pdf

A nice FAQ on what Metldown and spectre are plus a lot more information: https://meltdownattack.com/#faq-fix

Gary Williams

Gary Williams

IT Person | Veeam Vanguard | VMware vExpert | Windows admin | Docker fan | Spiceworks moderator | keeper of 3 cats | Avid Tea fan

Read More