I converted a Windows Server Update Services box to use HTTPS instead of the default HTTP port. Running on Server 2012 R2, this has most WSUS traffic use port 8531 instead of 8530.
I actually made this change a while ago but ended up reverting the change to HTTPS because I was unable to connect to the WSUS console afterwards. Giving it another shot, I followed the directions to secure WSUS with SSL to the letter again and I had the same results. When trying to connect to WSUS from the console on the server, I removed the previous connection and added a new server with the new URL and port 8531. I received the error message:
Cannot connect to ‘ServerName’. Please make sure the Post-Installation task is completed successfully in that server. If it was, please verify if the server is using another port or different Secure Sockets Layer (SSL) setting’.
I was fairly certain that I had completed the post-installation task, being WSUSUtil.exe configuressl <fqdn>. I did see a step mentioned that said to reinitialize the client proxy but I could never find any documentation that explained how to do that. I found a thread in Microsoft’s forums where someone asked the same question and was only harassed in response that they need to follow the directions.
As part of the change, the server path was changing from an internal fully-qualified domain name to a public facing domain name and the certificate from a public CA matched the new URL. I tested access to the WSUS files through a browser to ensure that clients were able to reach the server. I tested connecting to the WSUS console from another computer with the new information and it was surprisingly successful. With a little more searching and this new information, I found a known issue.
KB896861 explains a loopback security check that fails when the hostname does not match the local computer name. I was able to implement the workaround of specifying the alternative host names through the Registry.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
Name: BackConnectionHostNames
Type: Multi-string value
Data: public domain name
After creating that Registry entry and restarting the IIS Admin Service, the WSUS console suddenly started working properly from the WSUS server itself. Traffic of packages is secured over SSL and protected from man-in-the-middle attacks that had proof-of-concepts demoed last year.