Note from Microsoft:

"WSUS uses SSL for metadata only, not for update files. This is the same way that Microsoft Update distributes updates. Microsoft reduces the risk of sending update files over an unencrypted channel by signing each update. In addition, a hash is computed and sent together with the metadata for each update. When an update is downloaded, WSUS checks the digital signature and hash. If the update has been changed, it is not installed."

Thus, ports 8531 and 8530 must be open for communication for SSL on WSUS to work properly.

Steps demonstrating how to configure SSL on servers running the Windows Server Update Services.

1. Login to your WSUS server
2. Open up Server Manager        
3. Select Tools -> Internet Information Services (IIS) Manager 
4. Generate a SSL certificate
   1. Click on your Server and select Server Certificates 
      
   2. If you have your own PKI environment, follow these steps, if not, jump to step three
      1. Click 'Create Self-Signed Certificate' on the right side. (you can also create a domain certificate if you would like but it is not specified in this manual).
      2. 
      3. Fill in the field “Specify a friendly name for the certificate”.  Select the “Web Hosting” certificate from the drop down menu.  Click OK. 

      4. Open Sites in the connection tree > Click 'WSUS Administration'

      5. Under Actions column to your right, click on
      6. Select the 'https 8531' row and click edit
      7.  Select the SSL certificate you have just created in the dropdown list. Click 'View'

      8. Copy to clipboard the FQDN of the 'Issued to' server.  Click OK.

      9. Enter the hostname you have copied in the previous step to the Host name field. Click OK and then click close
         Note: Ensure the value you use in the Host name field is a FQDN. A ping from the Endpoint/s to the WSUS FQDN should be resolved to the correct IPv4 address
         
         
      10. 

      11. Under the 'WSUS Administration' tree click on 'ClientWebService' and then double click on the 'SSL Settings'.

      12. Mark the 'Require SSL' checkbox and then click Apply.

      13. Repeat the last two steps (9,10) for:
          1. 'DssAuthWebService'
          2. 'ServerSyncWebService'
          3. 'SimpleAuthWebService'. Close Internet Information Services (IIS) Manager. 
      14.  Start a command prompt in Administrator mode. 
          1. Change directory to C:\Program Files\Update Services\Tools. 
          2. Run WsusUtil.exe configuressl <FQDN>. 
          3. Make sure you get a similar URL response as seen in the screenshot.
          4. Close the command prompt.
      15. The next step would be to export the certificate. Run MMC in Administrator mode.  Click
      16. File>Add/Remote Snap-in
          
      17. Click on Certificates >  Click Add.
          
      18.  Click the radio button 'Computer account'.  Click Next.
          
      19. Click on the Finish button
          
      20. Click OK
          
          
      21.  Expand the Certificates (Local Computer) \ Trusted Root Certification Authorities and click on Certificates.  Right-click on the certificate that matches the FQDN of this server.  Click All Tasks > Export. The exported certificate can be used on WSUS client servers.
      22. Done!

Next Step

In order to enable secure communication between WSUS and the Windows endpoint you need to install the self-signed WSUS certificate on the endpoints.

See article