Hardening Windows Server 2003 SSL/TLS configuration

Though Windows Server 2003 has been around for a while, we'll still see them around the Internet for many years to come. Despite their usefulness, there are some important security considerations to make when running an Internet facing 2003 server.

SSL/TLS and the 2003 server
Windows Server 2003 was developed in a different era in terms of both security threats and the maturity of software security practices. In 2002, Bill Gates distributed his note on trustworthy computing internally at Microsoft. Since then Microsoft has made extensive efforts to produce more secure software — manifested in their SDL process. Among other tings, "secure by default" is an important principle in Microsoft's recent software. The 2003 server is not, so we'll discuss how to tighten up the SSL/TLS configuration. But first a quick primer on SSL/TLS.

The SSL/TLS protocols
SSL (Secure Sockets Layer), and its successor TLS (Transport Layer Security) are security protocols used to secure many types of Internet services, such as web, FTP, e-mail and so on. In their traditional use they ensure confidentiality and integrity for communication channel. SSL/TLS also offers server authentication in the most widely used setup. Find more information on the SSL/TLS protocols and their evolution on wikipedia.

SSL/TLS support is implemented in the schannel.dll in 2003 server. Many software packages rely on the native SSL/TLS support in Windows, e.g. Internet Explorer, the IIS (Microsoft's webserver software), and Google Chrome. It's therefore important to tighten up the SSL/TLS configuration.

To verify the current SSL/TLS configuration of a webserver, check out my previous blog post.

Disable weak encryption
A default configured 2003 server supports 40-bit encryption, and also the SSL 2.0 protocol. 40-bit encryption is subject to brute force attacks due to the short keylength. Why SSL 2.0 is supported is a mystery, as it was superseded by SSL 3.0 in 1996 due to critical security flaws.

Disabling the weak crypto requires registry changes, followed by a reboot. One can either take the tedious approach and edit the registry for hand, or the more efficient approach and put the required registry keys in a reg-file.

An example is included below, these are settings I have successfully loaded into registry on a Windows 2003 Server. A disclaimer is called for here, as editing the registry is not for the faint hearted. If something goes wrong the Windows installation might end up severely corrupted. Use the settings below at your own risk!

Copy the text (not including the horizontal lines), save it to a a file, e.g. sslsettings.reg, and double-click the file to load the settings into registry.

---

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Null]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 128/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"Enabled"=dword:00000000

---

After the settings are loaded to registry, a reboot is required for the changes to take effect. That's it! Any attempt to setup a SSL/TLS connection using SSL 2.0 or weak encryption will fail.

Mitigate MitM vulnerability
In November 2009 a vulnerability facilitating a Man-in-the-Middle attack on the SSL/TLS protocols was disclosed. Microsoft discusses the issue in a security advisory: SSL/TLS MitM vulnerability. Being the single most important security protocol on the Internet, TLS needed an update. Since 2009, various SSL/TLS implementations have been updated. Finally, last month (August 2010) Microsoft released an update through their Microsoft Update. The update implements RFC 5746, which solves the problem.

Add stronger crypto?
Since the 2003 server does not support AES cipher suites, there have been some interoperability issues with other systems. To cope with this, Microsoft released a hotfix adding support for the AES cipher. However, as the changes included in the latest TLS update are considerable, I would not risk installing the AES hotfix. Instead, consider upgrading to the 2008 server instead!