Optimizing the Windows Logon- and Logoff process
Transcription
Optimizing the Windows Logon- and Logoff process
Optimizing the Windows Logon- and Logoff process by Thomas Kötzing Optimizing the Windows logon- and logoff process ThomasKoetzing.de Notice The information in this publication is subject to change without notice. THIS PUBLICATION IS PROVIDED “AS IS” WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. THOMAS KOETZING, SHALL NOT BE LIABLE FOR TECHNICAL OR EDITORIAL ERRORS OR __ OMISSIONS CONTAINED HEREIN, NOR FOR DIRECT, INCIDENTAL, CONSEQUENTIAL OR ANY OTHER DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS PUBLICATION. Product names mentioned herein may be trademarks and / or registered trademarks of their respective companies. Copyright © 2004-2006 Thomas Kötzing, ThomasKoetzing.de, GERMANY. All rights reserved. DISCLAIMER No warranties of any kind. Use at your own risk. Version History Version Date __ Author Description 0.1 March 2003 Thomas Kötzing Initial version 0.2 June 2006 Thomas Kötzing Updated for Windows 2003 SP1 and Citrix Presentation Server Reviewed by Name Date Description Clemens Wunder March 2003 MCSE, CCEA Shane Broomhall June 2006 Novell CLE, CNE, CNI, CNS, CLP, RHCT, LPI1, Linux + LCP, LCI, CCEA, MCT, MCDBA, MCSE, MCSE + I, MCSA Jeff Pitsch July 2006 Microsoft MVP (2006) – Terminal Server, CCIA, CCEA, MCSE, HP ASE Page 2 Optimizing the Windows logon- and logoff process ThomasKoetzing.de My Intension - Your benefit Whenever you have problems with logging on- or logging off from Windows® Servers with terminal services in application mode, this document will help you. It will help with troubleshooting and optimizing the logon- logoff process. Your will find practical experiences written by Administrators for Administrators. This document applies to Windows 2000 and Windows 2003, with or without Citrix Presentation Server, and includes a detailed description of the related processes. The Author __ I have worked for a long time as Citrix Administrator and Freelancer on a lot of projects and in many different client sites. In 2003 I started posting in the official Citrix Support Forum where I have answered thousands of questions and I’m still one of the top users in the Citrix Forum. As a top poster I was invited by Citrix in November 2003 to attend an “Advanced Technical Training” in the EMEA Citrix Escalation Center Dublin. Later on Citrix invited me to be speaker at the Citrix Support & Engineering Institute of technology (CSEIT) 2004 in Orlando, where I have presented the first time about logon and logoff issues. Since then I was Presenter at BriForum 2005/2006, PubForum 2005, SBC-Solution Day 2005 and the CUG.no tech conference 2006. In February 2006 I moved from my old domain CITRIX4GE.DE to ThomasKoetzing.de and in July 2006 I was awarded as a Microsoft Most Valuable Professional (MVP) for Terminal Server. My thanks Thanks goes to my friend and business partner Clemens Wunder for his patience, enthusiasm and technical knowledge when working on the document as well for his support and impulses for my ideas. Bärbel Fischer, my dearest girlfriend for bringing the document in the right light, Shane Broomhall for correcting the English translation and Jeff Pitsch for reviewing the whitepaper. For their support, Jay Tomlin, Saul Gurdus and most of all Rene Alfonso (all from Citrix© Inc) as well as Robin Caron (Microsoft©). __ I like to hear your feedback regarding this document and I will try keeping it up to date. Thomas Kötzing Thomas@ThomasKoetzing.de While every effort has been made to ensure that the content of the document is accurate there are no warranties of any kind. Thomas Koetzing reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher. copyright 2004-2006, Thomas Kötzing, Nürnberg. All rights reserved. . Page 3 Optimizing the Windows logon- and logoff process ThomasKoetzing.de Index 1 Basics about logon and logoff – fast or slow............................................................................. 5 __ 1.1 Windows user profile.................................................................................................... 5 1.1.1 Local profile ........................................................................................................ 5 1.1.2 Roaming profile .................................................................................................... 5 1.1.3 Mandatory profile .................................................................................................. 5 1.1.4 Hybrid profile (FlexProfileKit).................................................................................... 5 1.2 Default profile folder structure ....................................................................................... 5 1.3 User registry hive........................................................................................................ 6 1.4 Understanding the logon and logoff process ......................................................................... 7 1.4.2 Realizations from the logon procedure ......................................................................... 9 1.4.3 Realizations from the logoff procedure ........................................................................10 2 Troubleshooting and Optimizing ........................................................................................ 11 __ 2.1 User profile folder redirection........................................................................................11 2.1.1 Profile analyzing ..................................................................................................11 2.1.2 Troubleshooting and optimizing.................................................................................11 2.1.3 Group Policy (Active Directory) .................................................................................11 2.1.4 Windows policies (NT Domain / Novell)........................................................................12 2.1.5 Logon script ........................................................................................................12 2.1.6 Deleting a roaming profile .......................................................................................13 2.1.7 The FlexProfileKit Version 5.0...................................................................................13 2.1.8 References for profiles ...........................................................................................13 2.2 User authentication ....................................................................................................14 2.2.1 Analyzing the logoff procedure .................................................................................14 2.2.2 Troubleshooting ...................................................................................................16 2.2.3 References for authentication ..................................................................................16 2.3 Logon scripts & start programs .......................................................................................16 2.3.1 Analyzing scripts & program starts .............................................................................17 2.3.2 Troubleshooting ...................................................................................................17 2.4 Citrix Client mappings .................................................................................................17 2.4.1 Analyzing CmStart - Client mappings...........................................................................18 2.4.2 Troubleshooting ...................................................................................................18 2.4.3 References to Citrix mappings ..................................................................................19 2.5 Server paths .............................................................................................................19 2.6 Group policy.............................................................................................................20 2.6.1 Analyzing of group policies ......................................................................................20 2.6.2 Troubleshooting ...................................................................................................20 2.6.3 References to group policies ....................................................................................20 2.7 Stuck profiles ...........................................................................................................20 2.7.1 Analyzing stuck profiles ..........................................................................................21 2.7.2 Troubleshooting & optimizing ...................................................................................21 2.7.3 References for hanging profiles .................................................................................22 2.8 Process UserInit with Windows 2003 Service Pack 1 and above..................................................23 2.8.1 References to UserInit with Windows 2003 SP1 ...............................................................23 2.9 Citrix Presentation Server Enhanced Logon Feedback ............................................................24 2.9.1 References to Citrix Enhanced Logon Feedback ..............................................................24 3 Citrix Presentation Server – Special case sessions remains active................................................ 25 3.1 Analyzing disconnected sessions......................................................................................25 3.2 Troubleshooting.........................................................................................................25 3.3 References...............................................................................................................26 4 Additional optimizations ................................................................................................. 26 4.1 Extending the timeout for the Registry- Flush operation .........................................................26 4.2 Starts the Desktop in it’s one process ...............................................................................26 Page 4 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 1 Basics about logon and logoff – fast or slow The duration of the logon and logoff process is often a subjective feeling for a user, especially if the user was just migrated from a Fat to a Thin Client. Otherwise the user will not notice that the new Thin Client might only need 5 seconds to bring up the logon screen. Nevertheless in real life there can be differences in the logon duration from seconds to 20 minutes or more (the times cannot be compared with a reconnection process). This is due to the fact that the logon process depends on the environment. Also by definition the Windows logon process starts when the secure attention sequence (winlogon window) shows up. 1.1 Windows user profile __ For a better understanding it’s useful to know the different profile types and the structure of the user profile. Microsoft supports three different kinds of Windows profiles. 1.1.1 Local profile The classic local profile is used basically on each Windows workstation. The profile is stored on the local system and applies only to that computer. 1.1.2 Roaming profile This profile type was introduced by Microsoft to support roaming users in the meaning of using multiple computers at different times. It was not introduced to support terminal service users but was the only solution at that time. The roaming profile is stored centrally and is copied and loaded during logon to the computer or server where the user tries to logon. This will guarantee a consistent appearance and behaviour in the user’s application and desktop settings. 1.1.3 Mandatory profile The mandatory profile is write protected and can be used as local or roaming profile. This is much more robust but clearly not very useful because the user settings are lost after logoff. __ 1.1.4 Hybrid profile (FlexProfileKit) This type of profile is a mixture of the mandatory and roaming profiles “found” by some smart consultants. Basically a mandatory profile is used but through logon and logoff scripts the user changes are stored in an extra file. 1.2 Default profile folder structure On the Server all profiles are stored in the default directory “Documents and Settings“ or in the folder that was defined in the Server registry. Registry System Key: [HKEY_LOCAL_MACHINE\ SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList] Value Name: ProfilesDirectory Data type: REG_EXPAND_SZ Value: %SystemDrive%\Documents and Settings (default) Microsoft Knowledge Base Article - 173870 Page 5 Optimizing the Windows logon- and logoff process ThomasKoetzing.de For the single user the profile structure looks like the following: __ Folder Name Description \Application Data Per-user roaming application data. \Cookies User Internet Explorer Cookies \Recent Shortcuts to recently used documents. \Desktop Desktop items, including files and shortcuts. \PrintHood Shortcuts to printer folder items. \Favorites Users Internet Explorer favorites. \Local Settings Temporary files and per-user non-roaming application data. \NetHood Shortcuts to My Network Places items. \SendTo Shortcuts to document storage locations and applications. \Start Menu Users personal start menu. \Templates Per-user customized templates. \Windows Windows system settings \My Documents Per users documents. 1.3 User registry hive When logging into a Windows system the user profile is loaded. What does that mean? As described before, the profile includes different subdirectories for different purpose and a file named ntuser.dat in the root of the profile (renamed to ntuser.man for a mandatory profile) This file holds all user specific registry settings. At login the ntuser.dat is loaded into the system registry hive HKEY_USERS with the Security Identifier (SID) of the user HKEY_USERS\<SID> __ Page 6 Optimizing the Windows logon- and logoff process ThomasKoetzing.de __ Illustration 1 In Illustration 1 the ntuser.dat of the local Administrator was loaded into the system registry. The important registry hives are Software, Control Panel and Environment. The SID ending with 500 is reserved for the local Administrator. After loading the user data into HKEY_USERS\SID the system creates a symbolic link to HKEY_CURRENT_USER. This process can be observed when opening the HKEY_USERS hive in the program regedit. With regedt32 the ntuser.dat can manually loaded into the registry as well as with regedit since Windows XP / 2003. 1.4 Understanding the logon and logoff process The following table explains the main logon and logoff procedure when connecting to a Windows Desktop. Points that differ from a Windows 2000 Server logon and logoff are marked differently. The listing of all needed steps already shows options for troubleshooting and optimizing. To get a better understanding, the main steps are described in detail __ In the brackets are the needed system processes and their calls to start new programs or processes. In the logoff process the UPHClean service was included. In the future a similar service will be part of the Windows operating system starting with Vista and Longhorn and the service will be updated through the Windows update. Those of you who want to know absolutely every detail of the logon process will find a great flow diagram under the following URL http://www.brianmadden.com/content/content.asp?ID=587 Page 7 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 01. Logon procedure The user starts the remote Windows desktop. 02. Only with Citrix Presentation Server Advanced or Enterprise In a Farm with load-balancing the client will be redirected to the least busy server. 03. The Microsoft licenses are checked. First the Client Access License (CAL) and next the TerminalService- Client- Access License (TSCAL) [Winlogon, Csrss] 04. The user is authenticated against the Domain (or local) and the session permissions are set. [Winlogon-> GINA (ctxgina->msgina/nwgina),Winlogon-> Lsass] 05. The System re-establish’s the user’s network connections. With Citrix Presentation Server the Enhanced Logon Feedback (stuisrv.exe) kicks in. [Winlogon-> UserInit, Winlogon -> stuisrv.exe] With Windows 2003 Service Pack 1 and above a second UserInit is launched and stays active for 1 minute for a certificate autoenrollment. [Winlogon-> UserInit] The System loads the user profile: From the Logon Server (Domain Controller) information is retrieved, if the user has a configured profile and what type: If so, then the system checks if a local profile already exists If so, then the system compares which profile is newer. If the remote profile is newer, then the profile is downloaded from the network share to the Windows server. If the user has no profile, then the system creates a new profile from the "Default User" and "All User". [Winlogon, UserInit-> Profile] The System applies all group policies configured for the user. [Winlogon, UserInit] The System reads the registry key "Shell" and starts the defined user shell. [Winlogon, UserInit->Explorer.exe] __ 05a. 06. 07. 08. 09. 10. __ 11. The System reads the "AppSetup" registry key and starts all programs that are listed in the key. (Witht Citrix Presentation Server CmStart.exe is added to the key). [Winlogon, UserInit-> UsrLogon.cmd, UserInit -> CmStart.exe] UsrLogon.cmd is executed to run possible compatibility scripts. [Winlogon, UserInit->UsrLogon.cmd] Only with Citrix Presentation Server CmStart.exe start’s the ICA Client Update Manager and the seamless engine wfshell. Part of wfshell is to open virtual channels (Client device mappings). Since PS4 the main Printer mapping is done the Citrix print service. [Winlogon, UserInit->CmStart.exe, CmStart->cltmgr.exe, CmStart->wfshell.exe] 12. The System start’s any Application that has been defined in the Group policy [Winlogon, Explorer->Applications] 13. The Explorer reads the "RUN" registry key and start’s all defined Application. [Winlogon, Explorer->RUN Keyl->Applications] 14. The Explorer start’s all programs from the "Autostart" directory of the users menu and from the AllUsers "Autostart". [Winlogon, Explorer->Autostart->Applications] Page 8 Optimizing the Windows logon- and logoff process ThomasKoetzing.de __ Illustration 2 Illustration 2 shows a screenshot of Process Explorer (from sysinternals.com) during a user logon on to a Citrix Presentation Server running Windows 2003. It shows the main processes that are involved at logon time. 1.4.2 Realizations from the logon procedure Copying big profiles It’s obvious that a roaming profile is copied to the terminal server each time a user logs on. Therefore the pure copy process of the profile can consume a good amount of the logon time. Domain Controller as bottleneck Depending on the Server Farm size and the number of users that have to be authenticated, the domain controller can slow down the logon process. The domain controller may not have enough performance to handle all the logon requests. Incorrect Autostart-Scripts A lot of programs and scripts can be executed during logon. If those programs start incorrectly or if scripts are not well programmed or even have minor errors, then the logon can slowed down again. __ Citrix / Microsoft Mappings Citrix and Microsoft client mappings can slow down the logon if a user has a fair amount of local or Network printers as well as client drives. Group policies A large amount of group policies can slow down the logon, since all policies need to be applied before the process continues. Stuck profile If a profile get’s stuck in the Server registry during logoff, it can result in slower logons and problems with the user profile in general. In a worst case scenario the user get’s a completely new default profile. Page 9 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 01. 02. 03. 04. 05. 06. __ 07. 08. Logoff procedure The user ends the Windows Terminal Server session. [Winlogon->Csrss] The System start’s the Applications that have been defined in group policies for logoff [Winlogon, Csrss, Explorer] The System terminates any process that are running in the user context after a defined timeout. [Winlogon-> Csrss] The System "tries" to unload the user registry hive from the server registry. [Winlogon, UserInit->HKCU\<SID>] The unload process is repeated as defined in the group policy, once a second for 60s by default. [Winlogon, UserInit] If the unload is not successful, then the UPHClean service kicks in. UPHClean closes any open registry handle the user registry hive and forces the unload of the user profile. [Winlogon, UserInit->UPHClean] The System copies the roaming profile back to the network share. [Winlogon->UserInit] The System deletes the local cached user profile, if that was defined in the group policy. [Winlogon->UserInit->Profil] 1.4.3 Realizations from the logoff procedure Copying big profiles The same as with logon. Big profiles extend the logoff time from a Windows system. Process termination The stopping of processes as well as the execution of programs and scripts can take some time before the logoff will continue. Unload of the user registry hive The repeated unloading of the profile can take up to one minute by default, before UPHClean will force the profile to be unloaded. Special case Citrix published application There is a special case where actually a logoff never happens because a sub process of the published application is still active. For the user it looks like a successful logoff. __ Page 10 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2 Troubleshooting and Optimizing The results from the logon and logoff showed that there are some potential points for optimization. The following optimizations can be a challenge to implement in a production environment 2.1 User profile folder redirection The part of the user profile that get’s copied between the network share and the server should be as small in size as possible. The Microsoft folder redirection group policy can redirect important profile sub directories to a network share and therefore reduce the total size of the user profile. __ 2.1.1 Profile analyzing Find out the size of the user profile folder by using Windows explorer or eventually the whole profile share for all users with a useful tool like treesize. Even if the profile has an acceptable size the logon might take too much time. The reason for this can be a large number of very small files within the user profile. Typically these small files reside in the folders: Cookies, Recent and Favorites. Copying hundreds of files with a size of 500 byte can take a lot of time. Using the free Microsoft Windows Resource Kit program "DIRUSE" will show the amount of files within the profile. 2.1.2 Troubleshooting and optimizing Folder redirection can be set manually through registry keys or configured in the group policies. Microsoft way is using the group policy “Folder Redirection”. The same effect can be reached in an NDS or Windows NT domain using scripts or Windows NT policies. With the use of folder redirection, directories from the user profile can be redirected to a different place. Within a Terminal Server environment the best place to redirect profile folder is the home drive of the user. When using folder redirection a highly available file share (cluster solution) should be used. If the user is still logged on to the Terminal Server and the connection to the file share is lost, then the system can’t reach the redirected folders. This will result in fatal errors for user applications or missing icons, program settings and so on. __ After activating folder redirection the defined directories are not copied anymore from the network share to the Terminal Server. This will reduce the size of the user profile and therefore can dramatically speed up the logon. Important folders for redirection are Application Data, Desktop and My Documents. Using scripting techniques, every profile sub folder can be redirected. 2.1.3 Group Policy (Active Directory) Group Policy Object (GPO) User Configuration: Only visible in Active Directory User Configuration\Windows Settings\Folder Redirection GPO Name: Application Data, My Documents, Desktop, Start Menu Status: Enabled Microsoft Knowledge Base Article - 232692 Page 11 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.1.4 Windows policies (NT Domain / Novell) With NT/NDS a system policy (ADM – File) needs to be created and is based on the scripting solution. Make sure that the policies do apply to user profiles. 2.1.5 Logon script During logon the following registry keys needs to be changed per user. This can be done for example with Microsoft Resource Kit program regini.exe (part of Windows 2000 and above) Registry __ User Key: [HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders] Value Name: Personal Value Name: My Pictures Value Name: Favorites Value Name: History Value Name: AppData Value Name: Desktop Data type: REG_EXPAND_SZ Value: %USERPROFILE%\Value Name (default) __ Illustration 3 In Illustration 3 all possible folder redirection registry keys are shown. The Active Directory group policy for folder redirection is limited for Application Data, Desktop, My Documents and the Start Menu whereas the scripting solution can redirect any folder that is defined in the User Shell Folders hive. Again, make sure that the changes apply to the user profile. Page 12 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.1.6 Deleting a roaming profile To keep the roaming profile consistent and to save storage space on the terminal server the cached roaming profile should be deleted after logoff. The automatic removal of the roaming profile is done though the Windows system when set in the group policy or the Server registry. Group Policy Object (GPO) Computer Configuration: Computer Configuration\Administrative Templates\System\Logon or with Windows 2003 Computer Configuration\Administrative Templates\System\User Profiles GPO Name: Delete cached copies of roaming profiles Status: Enabled __ Microsoft Knowledge Base Article - 274152 Registry System Key: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon] Value Name: DeleteRoamingCache Data type: REG_DWORD Value: 1 Microsoft Knowledge Base Article - 173870 2.1.7 The FlexProfileKit Version 5.0 The FlexProfileKit offers a lot of options to simplify profile handling and reduces the problems of logon and logoff issues. This profile variant is already deployed in big environments and has been proofed has a flexible and robust solution. Flex is also the right choice for profile migrations. The FlexProfileKit was “found“ by Jeron van de Kamp (to say it with his own words). The main component is a modified Office wizard from the Microsoft Office Resource Kit. The wizard is launched through logon and logoff scripts during logon. Not visible for the user the wizard reads or writes defined user registry keys to or from a separate file. The file is copied between the server and the file share but can automatically be compressed up to 7 times. Since the Administrator defines what keys should be saved, the user specific file is very small in size. __ In Terminal Server (Citrix) Farms the FlexProfileKit is highly recommended and has become a default installation in Server-based Computing environments. 2.1.8 References for profiles Step by Step Roaming Profiles Configuration http://support.microsoft.com/kb/161070 How To Change the Default Location of User Profiles and Program Settings http://support.microsoft.com/kb/322014 Folder Redirection feature in Windows http://support.microsoft.com/kb/232692 How to create a roaming user profile in Windows 2000 http://support.microsoft.com/kb/302082 Roaming Profile Folders Do Not Allow Administrative Access http://support.microsoft.com/kb/222043 Page 13 Optimizing the Windows logon- and logoff process ThomasKoetzing.de How to Automatically Delete Locally Cached Profiles http://support.microsoft.com/kb/173870 Using Group Policy to Delete Cached Copies of Roaming Profiles http://support.microsoft.com/kb/274152 Policies and Profiles Standards http://support.citrix.com/article/CTX19327 User Profile Deletion Utility (Delprof.exe) http://www.microsoft.com/downloads/details.aspx?familyid=901A9B95-6063-4462-8150360394E98E1E __ Directory Disk Usage (Diruse.exe) http://www.microsoft.com/downloads/details.aspx?FamilyID=955d7f2f-73d9-4018-9dd742da210e62ee User Profile Best Practices for MetaFrame Presentation Server http://support.citrix.com/article/CTX110351 Flex Profile Kit v5.0 http://portal.loginconsultants.nl/forum/attachments/FPKv5.0.zip 2.2 User authentication Often the problem in a terminal server environment lies in other components such as the authentication of the user against the domain. This is not a very common problem but if the server for authentication (AD, PDC, NDS) can’t handle a high number of requests during a short time, then the user logon will slow down since it must wait to be authenticated. 2.2.1 Analyzing the logoff procedure The logon process can be reviewed in great detail through a log file. The log will show timeouts and a lot of other information in the userenv.log file. The user environment verbose logging needs to be enabled per server. Registry __ System Key: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon] Value Name: UserEnvDebugLevel Data type: REG_DWORD Value: 10002 hex Microsoft Knowledge Base Article - 221833 Note: The activation of the verbose logging can be set on production servers without the need for a reboot. On the other hand the logging will slow down the logon and logoff process and therefore should be disabled after troubleshooting. The verbose logging is disabled simply by deleting the registry key. The log file can be found on the server in the following directory: %windir%\debug\usermode\userenv.log Page 14 Optimizing the Windows logon- and logoff process ThomasKoetzing.de __ Illustration 4 Illustration 4 shows a snapshot of the userenv.log file. On the left site you can find the time index that is logged in milliseconds. On 5:56:13:300 pm (17:56:13:300) the client was authenticated and then began the user profile loading. It’s obvious in what great detail the logon- and logoff process is monitored and a very good for troubleshooting. If the log file reaches a size of more that 300kb, then the log file will be renamed to userenv.bak when the server is re-started. Without a re-start the file can grow until the hard drive has no space left. __ Page 15 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.2.2 Troubleshooting If the logon problem is due to authentication then upgrading the logon server or adding additional logon servers will solve the problem. Another option to speed up the logon and is only indirectly related to authentication is to increase the number of the Terminal Server idle sessions. In times where many users try to logon not enough idle sessions might be available to handle the logons. This problem was eliminated with Windows 2003 and is handled dynamically. With Windows NT/2000 the number auf idle sessions can be changed through the registry where each additional idle session will consume 2MB RAM. Registry __ System Key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server] Value Name: IdleWinstationPoolCount Data type: REG_DWORD Value: 2 Microsoft Knowledge Base Article – 243215 Note: After changing the registry value a Server re-start is needed 2.2.3 References for authentication User Data and Settings Management http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/manageme nt/user01.mspx Examining the Terminal Server Key in KEY_LOCAL_MACHINE\System\CurrentControlSet\Control http://support.microsoft.com/kb/243215 How to Enable User Environment Debug Logging in Retail Builds of Windows http://support.microsoft.com/kb/221833 __ 2.3 Logon scripts and start programs There are a lot of options to starting scripts and programs during logon and logoff. Those options need to be examined. When Software is installed, programs are often added to the RUN registry key or shortcuts are copied to the StartUp folder and therefore executed for every user. A typical example is a quick start program in the system tray for the network card or a virus scanner monitor (for normal users?). Does the program XY need to be started for every user even though only a few actually need it? Can the logon script be optimized? You need to check if programs really need to be executed for every user at logon time. The program Autoruns from sysinternals.com shows quickly and graphically what programs and scripts are automatically launched during server boot or user logon. Page 16 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.3.1 Analyzing scripts and program starts To remove errors in login scripts you have to find them first. Especially with logon scripts that run seamlessly from the user, it’s difficult to find the issue. To troubleshoot scripts you should: __ Enable the display output (ECHO ON) Add pauses in the script for testing Redirect the display output into a log file. Remove any error that might happen through the script. Also non optimised scripts can slow down the logon. An example would be the extended use of the resource kit program iFmember. When enumerating more that 50 groups in the script with iFmember, then it will slow down the logon process. Registry System Key: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run] [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce] [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Microsoft\Windows\CurrentVersion\Run] [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Terminal Server\Install\Software\Microsoft\Windows\CurrentVersion\RunOnce] User Key: [HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Run] [HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce] Additionally check the programs in the Startup directory: %AllUsersProfile%\Start menu\Programs\Startup __ 2.3.2 Troubleshooting If scripts are used intensively a more effective scripting language than the windows shell script (CMD) should be used. Switching to VBS, WSH, KIX, MSH etc. scripts can minimize the batch processing. It’s not about one program or script but it’s the amount of all things that run during logon and everything should be analyzed to see if it’s really needed. It’s not only the slow down of the logon, it’s also the waste of server resources when applications are launched for every user. A good example is the Citrix single sign on service that is enabled by default when installing Citrix Presentation Server. The ssonsvr.exe process will launch in each session and consumes about 4MB per user and sometimes has a bad influence on the user logoff. If pass through is not used, then disable single sign on. 2.4 Citrix Client mappings The program CmStart.exe (Client Manager Starting Utility) is a Citrix process that is responsible for the start of the Client Manager to keep ICA Clients updated. In the end, CmStart fires up Citrix seamless engine wfshell.exe and is responsible for client mappings like printer, drives, COM, LPT etc. It’s also good to ask if every client printer needs to be mapped or if only mapping the default printer is good enough. Mapping all printers can slow down the login and the same applies to all other mappings. Also the windows print spooler is involved for printer mappings and is enumerating the status of all printers a user has. This is the reason why 3rd party printer driver can really be troublemakers for the logon process. Page 17 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.4.1 Analyzing CmStart - Client mappings With mappings you should keep an eye on the client. The amount of drives and printers that need to be mapped to a user session are important for the logon behaviour. Very important are the installed 3rd party print drivers. The ICA Client update is started through CmStart.exe and takes time to enumerate the local client version. __ 2.4.2 Troubleshooting As mentioned before, the automatic client update should be disabled when that Citrix feature is not used (Use another method of software distribution). With Presentation Server the client update can be disabled centrally through Citrix policies. Before Presentation Server the client update needed to be disabled manually per server via Start → run → cudutil.exe From the menu Database choose properties and then uncheck the Enable checkbox (Illustration 5) Illustration 5 To see if client mappings are the source for slow logon, disable, as a test, ALL client mappings using Microsoft or Citrix protocol configuration. __ Start → run → mfcfg.exe → ica-tcp → Client Settings → Client mappings overwrite → disable ALL options. Try the logon and re-enable step – by - step any needed mapping where you should enable the Windows Client printer mapping at last. Still a big problem is 3rd party print drivers. During logon the Microsoft print spooler subsystem is used to enumerate the status of each client printer If the print spooler service (spooler.exe) spikes CPU then it’s most likely because of a large number of client printers or a 3rd. party printer driver is not 100% compatible. Remove all 3rd party printer drivers and use only drivers that come with the original Windows CD. If you find not the right printer driver then use a compatible driver from the Windows CD. To find out what printer driver is the best choice use the printer driver matrix from http://www.printingsupport.com Citrix has changed the Network printer creation to Microsoft SMB since feature release 2 for MetaFrame XP 1.0 The printer are now mapped via SMB (Server message block) but is much slower as the previous Citrix client mapping method. With feature release 2 HotFix 102W2K065 and the following registry key is needed to turn back the way how Network printers are mapped. Page 18 Optimizing the Windows logon- and logoff process ThomasKoetzing.de Registry System Key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\ClientPrinterAutoCreate] Value Name: fCreateNetworkPrinter Data type: REG_DWORD Value: 1 (Network Printer as Client Printer=1, Feature Release 2 behaviour = 0) Citrix Knowledge Base Article - 101705 With MetaFrame XP 1.0 feature release 3 there is a new option in the Citrix Management Console (CMC) to revert this behaviour. You find the option in the properties of a published application under ICA Client options. Activate “Start Application without waiting for client printer creation” __ Another way to workaround the printer creation is to use CmStart.exe in the UsrLogon.Cmd where the script is executed asynchronously. To do so you need to remove CmStart.exe from the “AppSetup” registry key and place it in the UsrLogon.Cmd with the command Start "cmd.exe /c" "cmstart.exe". If now there is a problem with the printer mapping, then the logon will continue but the printers might be mapped later on. With Presentation Server 4.0 the previous workaround is not needed anymore, since Citrix introduced a print service CpsSrv that is now responsible for the client printer mapping. The new service should therefore prevent a hang or crash of wfshell and is also able to map the user’s printers after logon. 2.4.3 References to Citrix mappings What Does the CMSTART Command Do? http://support.citrix.com/article/CTX983798 Troubleshooting Citrix ICA Printer Autocreation http://support.citrix.com/article/CTX681954 Troubleshooting Slow Logons http://support.citrix.com/article/CTX101705 Run startup scripts asynchronously http://msdn.microsoft.com/library/default.asp?url=/library/en-us/gp/10.asp __ Presentation Server 4.0 Printing Enhancements http://support.citrix.com/article/CTX108170 2.5 Server paths It might sound trivial but check the server’s path environment variable. The variable should not have double entries and the important paths (System, System32, Citrix) should be at the beginning of the string (some application tend to place themselves at the beginning of the variable) During the ICA handshake the client’s available fonts are enumerated. During logon those fonts are searched on the Server and if the fonts are not in the server path it can take up to 20min. before the logon continues. This is the reason why Citrix recommends adding the font path to the server path variable: %SystemRoot%\fonts Page 19 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.6 Group policy In large domains with cascaded group policies, the logon can slow down to apply all the policies to the current user. With Windows 2000 there was a group policy to apply the policy asynchronously. 2.6.1 Analyzing of group policies To determine if the group policies are slowing down the logon, a review of the userenv.log will show if this is the case. The verbose logging needs to be enabled, see point 2.2. Find the first line that shows the start of group policies and notice the time index. Follow along the group policy to the last line and see how much time was consumed to apply all the group policies for the user. __ 2.6.2 Troubleshooting It is advisable to enable the group policy for loopback policy processing in replace mode. This will ensure that only the group policies for the OU are applied and all others are replaced. This will not only speed up the logon but will also ensure the security of the terminal server, since you “know” what is applied. Group Policy Object (GPO) Computer Configuration: Computer Configuration\Administrative Templates\System\Group Policy GPO Name: User Group Policy loopback processing mode Status: Enabled Modus: Replace Microsoft Knowledge Base Article - 231287 2.6.3 References to group policies Loopback Processing of Group Policy http://support.microsoft.com/kb/231287 Troubleshooting Group Policy Application Problems http://support.microsoft.com/kb/250842 __ How To Install and Use Resultant Set of Policy (RSoP) in Windows Server 2003 http://support.microsoft.com/kb/323276 How To Optimize Group Policy for Logon Performance in Windows 2000 http://support.microsoft.com/kb/315418 How to apply Group Policy objects to Terminal Services servers http://support.microsoft.com/kb/260370 2.7 Stuck profiles “Stuck“ profiles have a great influence on the logon behaviour. In best case the user registry hive is fully unloaded, the profile copied back to the network share and finally deleted on the terminal server. Under certain circumstances processes might have an open handle to the user registry hive at the time when a user logs off. An example is Windows installer service that accesses the user hive to maybe repair settings. If the system is not able to unload the user registry hive, then the hive remains in the server registry – it’s stuck. If the same user logs in again, several things can happen from getting a new profile to even being unable to logon. Page 20 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.7.1 Analyzing stuck profiles The unload misbehaviour is logged in the server event log as event ID 1000 “Windows cannot unload your registry file”. Additionally you can find the user SID in HKEY_USERS after the user has logged off and you still can’t unload the hive using regedt32. In this case only the termination of the open handle to the user registry hive or a server reboot will free the user registry hive. __ 2.7.2 Troubleshooting & optimizing To solve the user registry hive unload problem, Microsoft programmer Robin Caron has developed a service that closes any open handle and forces the unloading of the profile. The service is called UPHClean – User Profile Hive Cleanup and works very well. The version 2.x of UPHClean can also close open file handle to the user profile and fix other file or folder permission issues. With the next version of Windows Server (Longhorn) and Workstation (Vista) the UPHClean is included in the OS and will be automatically updated through the Windows update service. UPHClean should be installed on every terminal server to make sure the profile get’s unloaded. Before UPHClean kicks in the system retries the unloading of the user profile. The unload retries can be configured through group policies. Group Policy Object (GPO) Computer Configuration: Computer Configuration\Administrative Templates\System\Logon or with Windows 2003 Computer Configuration\Administrative Templates\System\User Profile GPO Name: Maximum retries to unload and update user profile Status: Enabled Max retries: 60 (default) Default is 60 retries once a second but with UPHClean installed the value can be reduced to a much lower value or even null. If a user logs off when he still has open applications, then the system has to terminate each running program. The system will grant each application a timeout to gracefully close itself. The timeout value can be set through the registry but be aware that if a user has an unsaved document it will not be saved and the user will also not be asked to save it. There are two registry values AutoEndTask and WaitToKillAppTimeout – with a default value of 20 seconds. __ Registry User Key: [HKEY_CURRENT_USER\Control Panel\Desktop] Value Name: AutoEndTask Data type: REG_SZ Value: 1 (Enables the automatic termination) Value Name: WaitToKillAppTimeout Data type: REG_SZ Value: 5 (Timeout of 5 seconds before the process is terminated) Page 21 Optimizing the Windows logon- and logoff process ThomasKoetzing.de A better way to set the user registry values is the global system registry key Registry __ System Key: Microsoft RDP [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDPtcp\UserOverride\Control] Citrix ICA [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\ICAtcp\UserOverride\Control] Value Name: AutoEndTask Data type: REG_SZ Value: 1 (Enables the automatic termination) Value Name: WaitToKillAppTimeou Data type: REG_SZ Value: 5 (Setzt eine Wartezeit von 5 Sekunden, bevor der Prozess beendet wird) With Windows 2003 those global keys don’t work anymore and has to be set on a per user basis. In some cases there are some “left over’s” in %UserProfile% that can be removed through the Windows Resource Kit Utility Delprof. Delprof can be executed as a Windows Task nightly or after a server reboot. 2.7.3 References for hanging profiles User Profile Hive Cleanup Service http://www.microsoft.com/downloads/details.aspx?FamilyId=1B286E6D-8912-4E18-B57042470E2F3582 Troubleshooting profile unload issues http://support.microsoft.com/kb/837115 Roaming User Profiles Do Not Unload http://support.microsoft.com/kb/253820 A roaming user profile does not upload successfully http://support.microsoft.com/kb/321936 __ Issues When Windows 2000 Loads and Unloads Profile http://support.microsoft.com/kb/289564 Roaming Profiles May Not Unload After You Install a New Printer http://support.microsoft.com/kb/829730 UsrClasses Hive Does Not Unload During Logoff Because of an Intermittent Handle Leak in Spoolsv.exe http://support.microsoft.com/kb/828153 Closing Timed-Out Applications Without Choosing End Task http://support.microsoft.com/kb/123058 Terminal Server and Connected Terminal Services Clients Pause When a Terminal Services Client Logs On or Logs Off http://support.microsoft.com/kb/324446 Page 22 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.8 Process UserInit with Windows 2003 Service Pack 1 and above With Windows 2003 Service Pack 1 Microsoft has moved the user and computer certificate autoenrollment into the Userinit process. The enrollment has up to 70 seconds to complete and during that time a logoff from a published application is not possible. If a certificate autoenrollment is not needed then it can be disabled through a per user registry key or group policies. Registry User Key: [HKEY_CURRENT_USER\Software\Microsoft\Cryptography\AutoEnrollment\AEExpress] __ GPO Computer Configuration: with Windows 2003 Computer Configuration\Security GPO Name: AutoEnrollment Status: Disabled Illustration 6 2.8.1 References to UserInit with Windows 2003 SP1 __ A remote session does not end immediately on a computer that is running Windows Server 2003 Service Pack 1 http://support.microsoft.com/kb/901196 Published Application Sessions Take 60 Seconds to Log Off when Windows 2003, Service Pack 1 is Installed http://support.citrix.com/article/CTX106049 Certificate Autoenrollment in Windows Server 2003 http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/au toenro.mspx Page 23 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 2.9 Citrix Presentation Server Enhanced Logon Feedback With MetaFrame Presentation Server 3.0 Citrix has introduced the “Enhanced Logon Feedback”. The purpose is to make the logon process more seamless for the end users. The feature “hides” running logon scripts, etc. and therefore displays the current process in the Enhanced Logon Feedback GUI (see illustration 7 with a customized feedback GUI). Basically the Enhanced Logon Feedback feature has only two processes, CtxHide.exe and stuisrv.exe. CtxHide.exe “hides” the command line window of scripts execution (UsrLogon.Cmd). The process stuisrv.exe is launched through winlogon.exe and is responsible for the transparent user logon feedback. __ Illustration 7 The Enhanced Logon Feedback can be the source of logon issues. To find out if the logon feedback is responsible for logon issues, it can be disabled through the following registry key: Registry System Key: [HKEY_LOCAL_MACHINE\Software\Citrix\Logon] Value Name: DisableStatus Data type: DWORD Value: 1 (Disables the Enhanced Logon Feedback) The process CtxHide.exe might be a problem as well (not so often). CtxHide.exe should also be disabled when an Administrator wants to troubleshoot the logon script. If there is something not working well in the UserLogon.Cmd, then the Administrator has no visible feedback about it. With a temporary removal of CtxHide.exe from the AppSetup key, the Administrator will see once again the execution of the logon script. If CtxHide.exe is the troublemaker, then it can be replaced with other freeware tools like cmdow, runh etc. __ Registry System Key: [HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\Currentversion\Winlogon] Value Name: AppSetup Data type: DWORD Value: CtxHide.exe UsrLogon.cmd,CmStart.exe (remove CtxHide.exe) 2.9.1 References to Citrix Enhanced Logon Feedback Explaining CtxHide.exe and hiding UsrLogon.cmd http://support.citrix.com/article/CTX104352 Disabling the Enhanced Logon Feedback for ICA Client http://support.citrix.com/article/CTX104893 Page 24 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 3 Citrix Presentation Server – Special case sessions remains active After a user has ended a published application (not a Desktop) the Window is closed on the user side but the session remains active on the server. This is due to additional programs that have been started or issues with the seamless engine (wfshell). As a result the user is still logged on to the server and a logoff never happened. Another login of the same user might not work (session limit set) or might end with profile issues. The main reason for this is that additional programs are started without the knowledge of the seamless engine, for example virus monitors, sms client, single sign on, notes demons and so on. When a published application is closed the seamless engine (wfshell) will only close the programs that where started through wfshell; any additional programs remain active and stop the logoff process from completing. 3.1 Analyzing disconnected sessions __ You will notice the problem in the Citrix Management Console (CMC) or user problems to logon when session limits are in place. Reviewing the sessions will show a bunch of application as disconnected (most likely for the same published application). Through the CMC or the task manager, the still open processes are visible for the disconnected sessions. Illustration 7 3.2 Troubleshooting __ The main target is to figure out which additionally started process prohibits the graceful logoff from the server. Use the CMC to open the session properties (or with task manager on the server) of the stuck session and terminate the processes (not winlogon.exe and csrss.exe) one-by-one until a graceful logoff occurs. Once you figure out which process frees up the logoff (let’s say it’s “xyz.exe”) think about whether you really need that executable. If not, then delete it or uninstall the application. If the executable is important then you can add a seamless engine registry flag (LogoffCheckSysModules) to include the process for termination through wfshell. The seamless engine reads the key at logon time and therefore the setting is active with the next logon. Here an example fort he program xyz.exe Registry System Key: [HKEY_LOCAL_MACHINE\ SYSTEM\CurrentControlSet\Control\Citrix\wfshell\TWI] Value Name: LogoffCheckSysModules Data type: REG_SZ Value: xyz.exe (additional programs separated through comas. Example: App1.exe,App2.exe,App3.exe) Find a detailed description in Citrix Knowledge Base Article CTX891671 Page 25 Optimizing the Windows logon- and logoff process ThomasKoetzing.de 3.3 References Graceful Logoff from a Published Application Keeps Sessions in Active State http://support.citrix.com/article/CTX891671 Seamless Exception Registry Flags http://support.citrix.com/article/CTX101644 4 Additional optimizations There are a lot of other sources that can result in errors or slow down of the logon and logoff process that comes from additional Software (Novell, SmartCard etc.), Service Packs, Hotfixes and so on. Search all the related Software vendors Knowledge Bases about potential problems. __ The following registry keys have shown to be helpful with the logon- and logoff process. 4.1 Extending the timeout for the Registry- Flush operation Registry System Key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Session Manager\Configuration Manager] Value Name: RegistryLazyFlushInterval Data type: REG_DWORD Value: 5 (Value should be between 30 and 60) 4.2 Starts the Desktop in it’s one process Registry System Key: [HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer] Value Name: DesktopProcess Data type: REG_DWORD Value: 1 __ Page 26