Today I wanted to mix it up a little. I want to talk a little about debugging, more specifically about using driver verifier. If you have seen a Windows bug check, aka the blue screen of death, then you have probably seen a DRIVER_IRQL_NOT_LESS_OR_EQUAL. This particular bug check has two codes associated with it; 0x0000000A and 0x000000D1. In the majority of cases, this bug check is caused by a 3rd party driver that is misbehaving. Most of the time the cause can be identified by debugging the kernel memory dump. Sometimes however the problematic driver is paged out and is no longer present in the memory dump. In this case we can use driver verifier to catch this misbehaving driver and force a bug check to happen before the driver is paged out. The rest of this article will discuss how to enable driver verifier for this scenario.
- Click Start, or in Windows 2012 and higher right-click Start, and choose Run. Type in verifier and click OK.
- From the select a task options, select “Create custom settings (for code developers)” and click Next.
- From the select individual settings from this full list, tick the check box for Special pool and click Next.
- From the select what drivers to verify options, select “Select driver names from a list” and click Next.
- Sort the driver list by provider. Tick the check box for all drivers that are not provided by Microsoft, and click Finish. We do this so that we can monitor all 3rd party drivers.
- Reboot the server.
After the server is rebooted all 3rd party drivers will be monitored. If one of them should misbehave, driver verifier will force the system to bug check. It should then be fairly simple to find the problem driver in the memory dump.
If you have any suggestions for this article, please leave a comment below.
Good morning. I figured it was time for another post on Essentials. Some parts of this article also apply to Foundation edition.
Let’s start off with a little background on Essentials edition. Windows Server Essentials edition is designed for a small to medium sized business. It is a very good option for a small to medium sized business with less than 25 users/computers. Here are a few of the advantages to running Essentials.
- It is less expense than standard edition. Typically by $200-300.
- There are no additional CALs (Client Access Licenses) to purchase. Twenty five user CALs are included.
- Can be easily upgraded to standard edition with a single command.
- Client PC Backup is builtin. This feature automatically backs up client PCs to the server
- Anywhere access is available. This is a feature that was first introduced in SBS (Small Business Server) It allows a user to remotely access computers and file shares. It also allows the administrator to access the Dashboard from anywhere. Additionally the administrator can setup a SSTP (Secure Socket Tunneling Protocol) VPN (Virtual Private Network) via a wizard.
- Easy integration with Microsoft cloud solutions.
What’s the catch? There is always a catch. Windows Essentials edition is no exception. Below are the limitations imposed by running Essentials.
- The Essentials server MUST be a domain controller.
- The Essentials server must hold all the FSMO (Flexible Single Master Operation) roles. If you want to learn more about the FSMO roles here is a good article.
- Only one domain is permitted in the forest where the Windows Essentials edition server resides.
- No forest/domain trusts are permitted.
- The Remote Desktop Session Host role feature is not supported and typically will not function.
So what if the server is not a domain controller or violates one of the rules above? This is where the Server Infrastructure License Service comes into play. This service regularly checks the server to verify it is not violating the EULA (End User License Agreement). If a violation is found the server will shutdown every 27.67 days (27 days, 16 hours). Why Microsoft chose 27.67 days, I have no idea. Before it shuts down though it will warn you. The events will show up in the Server Infrastructure Licensing log.
The next question is how do we fix these errors. I have seen three causes for this issue. Let’s go over each one and how to fix it. After you believe you have fixed the issue, see the the next section for a way to confirm the issue is resolved.
- The first cause of this issue is also the most painful to fix. If the server is demoted, and put into a workgroup it will cause this issue. All checks will fail because the domain can no longer be contacted. Unfortunately the only fix is to reinstall Windows on the server.
- The second reason these errors might crop up is due to the check failing due to an issue with Active Directory. For instance, if the server is not advertising as a domain controller due a SYSVOL issue. If an Active Directory issue is suspected, the first place to start should be to run a dcdiag. Dcdiag will test the basic functionality and report any issues found. As stated above, if there are SYSVOL issues, then the server will likely fail the advertising test.
- The last reason I have seen on more than a few occasions is the following error:
Log Name: Microsoft-Windows-Server Infrastructure Licensing/Operational
Event ID: 2
Description:The Forest Trust Check in the Licensing component did not pass because error 0x80070008 occurred in function fe1 [YJBI].
Not enough storage is available to process this command.
This error seems to indicate that we are low on hard drive space. However this is not the case. This error is actually referring to a special pool in memory (RAM, Random Access Memory) called the heap. The heap is a finite size, regardless of how much RAM is in the system. Normally Windows will not experience a heap exhaustion, that is where this special pool of memory is completely depleted. However, if a program or driver is leaking memory, then the pool will eventually run out. In case you are wondering what a memory leak is, it occurs when a program or driver allocates memory, but does not free it when complete.
So in essence this error is caused by a malfunctioning program or driver. The good news is that every time I have seen this issue in Essentials or Foundation it was caused by a printer driver. There are 2 ways to fix this problem. The first way is to simply restart the printer spooler service. Restarting the printer spooler service unloads the printer drivers and frees all memory associated with them. This will temporarily eliminate the issue. A scheduled task could then be created to automatically do this on a regular basis. The optimal solution though is to find the problem driver and either remove or update it.
So to this point we have covered some of the pros and cons of running Essentials, what happens when the EULA is violated, and some common causes for the Server Infrastructure Licensing service shutting down the server. The last item I wanted to cover is how to force a new compliance check from the Server Infrastructure Licensing service. This process works for both Essentials and Foundation edition. This is useful if you are seeing compliance check errors, have taken measures to correct them, and now want to test if the issue is resolved. It is surprisingly easy to force a compliance check. Only one PowerShell command is required. Ensure you run PowerShell as administrator when running this command.
Stop-Process -ProcessName silsvc -Force
The above command forces the Server Infrastructure License service process to stop. The process will then immediately start again. The trick here is that the Server Infrastructure License does a compliance check every time it starts. You should see a compliance check within 2-3 minutes after the service stops.
Well, we covered a lot of ground with this post. If you have any questions, or any suggestions please add a comment below.
In the last few articles I walked you through setting up RDS (Remote Desktop Services) in a domain, or a workgroup, and installing and activating CALs. Today I wanted to touch on how to fix RDS when it has been improperly deployed. The good news, it is typically very easy.
Typically when RDS is deployed improperly the user has installed the Session Host manually via the add roles wizard. While this is fine in a workgroup, it does not work properly in a domain environment. If this is the case, just running the Remote Desktop Services installation wizard is typically enough to fix the deployment. Just follow this guide. I have had about 90% success rate with this.
But what if the Remote Desktop Services installation wizard fails or does not complete? Don’t fret. More than likely the issue can still be resolved without removing the roles. Below are the two most common issues that I have run into when fixing an RDS deployment.
- The Remote Desktop Services installation wizard fails to create the Session Collection and remote apps. Usually when this happens it is because the user has attempted to fix the installation with Group Policy. There are two legacy group policy settings for specifying the license server and license mode. These settings are not supported in 2012, do not work, and will cause the Session Collection wizard to fail. The fix is simple, remove the group polices. Set them to Not configured. Here is a screen shot of what the policies should look like and their location.
Computer Configuration -> Administrative Templates -> Windows Components -> Remote Desktop Services -> Remote Desktop Session Host -> Licensing
- The Connection Broker role service role fails to deploy and the Remote Desktop Services installation wizard indicates the server requires a restart. Rebooting the server and running the wizard will again indicate a reboot is required.
The first time I ran into the this issue it took me a bit of time to track it down. After rebooting the server and checking for the file C:\windows\winsxs\pending.xml, and not finding it. Additionally, checking the c:\windows\log\cbs\cbs.log and seeing that a reboot is not pending, I was confused. I then dug deeper into the cbs log and found that the installation was being rolled back and that roll back required a restart. So why was the install failing? The key is an error logged in the system log during the install.
Log Name: System Source:
Service Control Manager
Event ID: 7041
Description: The MSSQL$MICROSOFT##WID service was unable to log on as NT SERVICE\MSSQL$MICROSOFT##WID with the currently configured password due to the following error: Logon failure: the user has not been granted the requested logon type at this computer.
Service: MSSQL$MICROSOFT##WID Domain and account: NT SERVICE\MSSQL$MICROSOFT##WID
This service account does not have the required user right “Log on as a service.
The Connection Broker service requires a SQL database to store its data. By default it uses the WID (Windows Internal Database). Because the WID cannot start, the Connection Broker role service install fails and the install has to be rolled back. This typically only occurs when the RDS server is also a domain controller. It is possible though on a member server if security has been tightened. The fix is fairly easy and is documented in Microsoft KB 2832204. Because this typically occurs on a domain controller though, you cannot simply edit the local group policy or the local security policy. You must create a new GPO (Group Policy Object) and link it to the domain controllers container. Here is the process.
- On a domain controller, launch the group policy management console. (gpmc.msc)
- Expand Forest, Domains, your domain name, then Domain Controllers.
- Right-click on the Domain Controllers container and choose Create a GPO in this domain and Link it here.
- Give the GPO a descriptive name like “Allow WID logon on DC” and click OK.
- Right click the new GPO and choose Edit.
- Navigate to the following location: Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment
- Right-click on Log on as a service and choose Properties.
- Check the box to Define these policy settings.
- Click on Add User or Group and add the following user: NT SERVICE\ALL SERVICES
- Click OK and close the Group Policy Management Editor window.
- To make the setting take effect immediately, run gpupdate /force at an administrative command prompt.
If the above solutions did not resolve the deployment issues with RDS, there is another option. If all else fails, remove all RDS role features* and start the deployment over again. As bad as this sounds it typically only takes 30-60 minutes to remove the roles and redeploy correctly.
If I missed anything, please submit a comment below. Thanks for reading.
* If Licensing has been configured and RDS CALs (Client Access Licenses) added, it is typically best to not remove the licensing role feature. That is unless you enjoy calling the Microsoft clearinghouse. 🙂
I have previously written an article on setting up RDS (Remote Desktop Services) in domain environment. However, what if there is only one server and/or there is no domain? It is still possible to setup RDS, but the process is a little different. Connection broker, a component of RDS, does not work properly in a workgroup environment. Also it is not possible to use Remote App, another component of RDS, or create a session collection. Without a session collection, Session Host, yet another RDS component, will not be able to obtain licenses. Without licenses RDS will cease to function after the initial grace period. Typically this is 120 days. We get around this by manually editing the WMI attributes for Session Host.
Here is how to properly deploy RDS in a workgroup. I am going to use PowerShell to make this quick and easy. This guide is applicable for Windows 2012 and higher in a single server workgroup environment.
- Launch a Windows PowerShell using the Run as Administrator option.
- Run the following command to install the RDS components that are required. (This will automatically restart the server)
Install-WindowsFeature RDS-RD-Server,RDS-Licensing -IncludeManagementTools -Restart
- After the server restarts, log back in.
- Launch a Windows PowerShell using the Run as Administrator option.
- Run the following commands to manually configure Session Host*.
- $obj = gwmi -namespace “Root/CIMV2/TerminalServices” Win32_TerminalServiceSetting
The next step is to activate the RDS license server and install RDS CALs. I will update this article with a link to that guide when it is available.
* On the $obj.ChangeMode() command:
$obj.ChangeMode(4) = Per User licensing
$obj.ChangeMode(2) = Per Device licensing
I have seen so many deployments of RDS (Remote Desktop Services) done incorrectly in Windows Server 2012/2012R2 that I wanted to get the correct process out there. Unlike previous versions of Windows, RDS in 2012 must be deployed with the RDS installation wizard. There is one exception to this rule. When there is only one server and it is in a workgroup, the deployment works differently. I will cover that deployment in another guide.
This guide is applicable for a Windows 2012/2012R2 server that is joined to a domain or is a domain controller. Following the below steps will take you through the setup of the first RDS server.
- Join the server to the domain. If the server is also to be the domain controller*, install the ADDS (Active Directory Domain Services) role and promote the server.
- Launch the Add Roles and Features Wizard in Server Manager.
- Launch Server Manager
- At the top right, click Manage
- Choose Add Roles and Features
- On the Installation Type screen, choose Remote Desktop Services installation and click Next.
- On the Deployment Type screen, choose Quick Start and click Next.
- On the Deployment Scenario screen, choose Session-based desktop deployment and click Next
- On the Server Selection screen, verify the server is selected and click Next.
- On the Confirmation screen, check the box to Restart the destination server automatically if required and click Deploy. The server will automatically reboot.
- Log into the server after the reboot. The wizard will automatically launch and the deployment should complete. Close the wizard when complete.
- Go back to Server Manager. On the left side click on Remote Desktop Services. You should see a screen similar to the following.
- Click the + symbol above RD Licensing. This will launch the wizard to add a licensing server.
- Add the server to the Selected list and click Next.
- Click Add. When the process completes, click Close
- Click the TASKS button next to DEPLOYMENT OVERVIEW and choose the option to Edit Deployment Properties.
- Click the RD Licensing option on the left side. Choose the licensing mode that matches the RDS CALs purchased. The licensing server should already be populated. If it is not type in the FQDN (Fully Qualified Domain Name) and click Add. Click OK.
You now have a very basic setup of Remote Desktop Services. The next step is to activate your license server and install your RDS CALs. I will write an article on that process in the near future and update this article once it is published.
*Microsoft discourages making an RDS server a domain controller. If it is required, see this technet article.
Here is a situation I run into on a frequent basis.
- We have a new server from a OEM (Original Equipment Manufacturer) like Dell, HP or Lenovo.
- The server has a factory installed OEM operating system.
- The server is setup as a Hyper-V host with virtual machines
- The user setting up the virtual machines does not have physical access to the server.
- The user extracts the product key from the host with a tool like Belarc advisor or Magical jellybean keyfinder.
- The user attempts to use the extracted product key on the virtual machines and activation is not successful.
So where did the user go wrong in the above situation? And if the product key works on the Hyper-V host, why will it not work on the virtual machines?
Before we can answer the above questions we need to understand a little about the type of product key used by large OEMs. Typically when a Windows operating system is installed by an OEM, an OEM_SLP (System Locked Pre-installation) key is used. This type of key is used to save time when building servers, as the server does not have to be activated with Microsoft. When this type of key is entered into a Windows installation, Windows will query the SLIC (System Licensed Internal Code) table located in the BIOS (Basic Input Output System) of the machine. If the table matches what is expected, then Windows will activate. With this in mind, a Dell OEM SLP product key can only be used on a Dell server. And an HP OEM SLP product key can only be used on a HP server.
To answer both questions. As explained above the product key in the Hyper-V host is an OEM_SLP key. This key requires that Windows query the bios of the system and verify the SLIC table matches. A virtual machine has an emulated bios from Hyper-V. The SLIC table does not match and Windows will not activate.
Now that we understand the problem, how can we identify it.
- A couple of questions to start with. Did the server come pre-installed with Windows. If so, has the product key been changed? If the server came pre-installed with Windows and the product key has not been changed, then we likely have an OEM_SLP product key.
- On the Hyper-V host and the virtual machine, do the following.
- Open an administrative command prompt
- run the following command: slmgr /dlv If the Product Key Channel is OEM:SLP, then you have confirmed we have an OEM:SLP product key
The solution is quite simple. The user needs to locate the product key sticker on the server and use the product key from it to activate the virtual machines. Here are the commands to run, from an administrative command prompt, on the virtual machine to do this quickly:
- slmgr /upk
- slmgr /ipk xxxxx-xxxxx-xxxxx-xxxxx-xxxxx (replace the x’s with the actual key)
- slmgr /ato
I ran across an interesting issue on Windows 2012 the other day, that I figured I would share. I had a customer that had 4 virtual machines on the same host that were all indicating they were not genuine. Aside from this we had 2 other strange behaviors.
- Server Manager would not populate any of the local server information fields. All fields showed Unknown.
- Slmgr commands were failing with “Error:0x80070422 On a computer running Microsoft Windows non-core edition, run ‘slui.exe 0x2a 0x80070422’ to display the error text.”
When running the command slui.exe 0x2a 0x80070422 we get the following description. The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.So which service is it talking about?
The Software Protection service. This service runs in the background and checks every minute or so whether the OS is licensed properly. In this particular case someone had disabled the sppsvc (Software Protection service). The fix was easy, re-enable the service and start it.
This brings me to another issue I have seen with sppsvc on several occasions. What if the service cannot be started due to an Access Denied error. This can typically be fixed, but we need to know where the Access Denied is coming from. The best tool, I have found, to do the job is Process Monitor. With this tool it is possible to see every time the filesystem or registry is touched and which process did it.
Here is how you setup Process Monitor.
- First download the tool here.
- Run procmon.exe from the zip file.
- Accept the EULA.
- Stop capturing (Ctrl+E) and Clear the display (Ctrl+X)
- Create a filter (Ctrl+L) with the following attributes. Result is ACCESS DENIED then Include. Add the filter and click OK. See the picture below.
6. Start capturing events in Process Monitor (Ctrl+E) and immediately start the Software Protection service.
7. As soon as the Access Denied error pops up, stop capturing events in Process Monitor.
At this point there should be one or more ACCESS DENIED events. When I have seen this issue, it has always been 2 registry keys that are missing permissions; HKLM\SYSTEM\WPA and HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform.
To correct the access denied issue the registry permissions will need to be restored. Normally that would mean restoring the permission for the account starting the service, in this case Network Service. The Software Protection service is a little different though. There is a special account that needs to be added, “NT Service\sppsvc”. Below are the special permissions that need to be granted on the registry keys.
- HKEY_LOCAL_MACHINE\SYSTEM\WPA. All subkeys need to inherit these permissions.
- Query Value
- Set Value
- Create Subkey
- Enumerate Subkeys
- Read Control
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\PayloadOverride