Perf OS Event ID 2012 - Citrix -Not accepting connections after 30 users

In citrix presentation Server 4.5/5.0 environment, the Citrix servers were rejecting user logons/sessions after certain number of active sessions(30 in my case). You can see errors in event log as mentioned below:
Event ID 2012 perfOS
Unable to contact server or XML broker
Server is busy to process request

There two reasons for the above scenario:
1. XTE Service -(session relaibility) -XTE service's "Non Paged Pool" size reaches 4 GB  that means you have some issue with memory. You can check this in task manger by enabling non paged pool in filter.

Solution: Apply the Microsoft hotfix http://support.microsoft.com/kb/936655

Apply the Citrix patch: http://support.citrix.com/article/CTX123956


2. If Free PTEs were less than 50K, then you can see above behavior:
Here is the solution:

Problem: ** Virtual Memory Usage ***

        Physical Memory:     2096923 (   8387692 Kb)

        Page File: \??\C:\pagefile.sys

          Current:  12288000 Kb  Free Space:  11893324 Kb

          Minimum:  12288000 Kb  Maximum:     13312000 Kb

        Available Pages:     1571173 (   6284692 Kb)

        ResAvail Pages:      1995682 (   7982728 Kb)

        Locked IO Pages:         245 (       980 Kb)

        Free System PTEs:        742 (      2968 Kb)

        ********** Running out of system PTEs **************

        ******* 809982 system PTE allocations have failed ******

Reason & Solution:

1.       System is booting up with about 50K PTEs. Usually the system should be have initial PTE of about 190K+. System has lost about ~140K PTEs (573 MB of Kernel VA). The system supports "Hot-Add Memory" feature which allows OS detect additional RAM which gets plugged in without needing a reboot. To allow this OS will reserve the Kernel VA to support the maximum RAM that can be added to the system. BIOS is responsible for notifying the amount of the maximum RAM that the system can support. In this case the BIOS notified the system that it can support up-to 128 GB of RAM but only 8 GB of RAM is currently plugged into the system. Hence OS reserved about ~880 MB of Kernel VA for the PFN DB expansion in case additional 120 GB of RAM gets plugged into the system.

2.       This behavior of OS can be controlled via the registry key documented in the article below, where we can specify the amount of RAM that OS should anticipate and reserve space for. Usually we set this value to 1.

913568        Decreased performance, driver load failures, or system instability may occur on Hot Add Memory systems that are running Windows Server 2003

http://support.microsoft.com/default.aspx?scid=kb;EN-US;913568

Recommendations:

================

1.       Set the Registry value documented in the Workaround section of the above mentioned KB article to 1 and reboot the system and the observe what is the value of Free System PTEs at the boot.

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

REG_DWORD DynamicMemory 0x1

 

2.       Also set the SystemPages value to 0 (ZERO)

 HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

REG_DWORD SystemPages 0x0

 

I am not repsonsible for any failures while applying above fixes.

Have a nice day!