OS: Server 2003 SP2
Hardware: Proliant DL580 x86 Processor
I am having a problem in which our 2 node file cluster has initiated "Failover" at least once a week. This cluster has been in place for 3yrs now and has normally been solid. We have not installed any new hardware/software.
Prior to failover I am getting the following System events:
1. Cluster File Share resource "name" has failed a status check. The error code is 1450
2. Cluster resource "name" in Resource Group "name" failed
Right before the 2 events above I usually get an event from the SRV (Server service) "The server was unable to allocate from the system paged pool because the pool was empty".
I get several of these events related to paged memory pool and was reporting this to be the cause of the cluster failover but I am not sure I am correct on that assumption. I decided to research the memory pool issues and have made some registry changes related to it as referrenced at
http://blogs.msdn.com/b/ntdebugging/archive/2006/12/18/understanding-pool-consumption-and-event-id_3a00_--2020-or-2019.aspx
http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101
So, what is happening is that several status checks fail and the resources begin to fail until a complete "Failover" occurs to the other node. I have not been able to identify anything in Cluster logs. Due to character limits I have attached portion of the logs for review. I would appreciate some help on discovering the problem.
*********************************************
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumSendUpdate: Dispatching seq 698413 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumSendUpdate: Locker updating seq 698413 type 0 context 11
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumSendUpdate: completed update seq 698413 type 0 context 11
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698413) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698414 Generation=0
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumSendUpdate: Locker dispatching seq 698414 type 0 context 11
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumSendUpdate: Dispatching seq 698414 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumSendUpdate: Locker updating seq 698414 type 0 context 11
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumSendUpdate: completed update seq 698414 type 0 context 11
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698414) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698415 Generation=0
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumSendUpdate: Locker dispatching seq 698415 type 0 context 11
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumSendUpdate: Dispatching seq 698415 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumSendUpdate: Locker updating seq 698415 type 0 context 11
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumSendUpdate: completed update seq 698415 type 0 context 11
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698415) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698416 Generation=0
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumSendUpdate: Locker dispatching seq 698416 type 0 context 11
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumSendUpdate: Dispatching seq 698416 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumSendUpdate: Locker updating seq 698416 type 0 context 11
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumSendUpdate: completed update seq 698416 type 0 context 11
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698416) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698417 Generation=0
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumSendUpdate: Locker dispatching seq 698417 type 0 context 11
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumSendUpdate: Dispatching seq 698417 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumSendUpdate: Locker updating seq 698417 type 0 context 11
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumSendUpdate: completed update seq 698417 type 0 context 11
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698417) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698418 Generation=0
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumSendUpdate: Locker dispatching seq 698418 type 0 context 11
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumSendUpdate: Dispatching seq 698418 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumSendUpdate: Locker updating seq 698418 type 0 context 11
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumSendUpdate: completed update seq 698418 type 0 context 11
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698418) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698419 Generation=0
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumSendUpdate: Locker dispatching seq 698419 type 0 context 11
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumSendUpdate: Dispatching seq 698419 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumSendUpdate: Locker updating seq 698419 type 0 context 11
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumSendUpdate: completed update seq 698419 type 0 context 11
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698419) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698420 Generation=0
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumSendUpdate: Locker dispatching seq 698420 type 0 context 11
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumSendUpdate: Dispatching seq 698420 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumSendUpdate: Locker updating seq 698420 type 0 context 11
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumSendUpdate: completed update seq 698420 type 0 context 11
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698420) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698421 Generation=0
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumSendUpdate: Locker dispatching seq 698421 type 0 context 11
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumSendUpdate: Dispatching seq 698421 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumSendUpdate: Locker updating seq 698421 type 0 context 11
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumSendUpdate: completed update seq 698421 type 0 context 11
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698421) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698422 Generation=0
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumSendUpdate: Locker dispatching seq 698422 type 0 context 11
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumSendUpdate: Dispatching seq 698422 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumSendUpdate: Locker updating seq 698422 type 0 context 11
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumSendUpdate: completed update seq 698422 type 0 context 11
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698422) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698423 Generation=0
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumSendUpdate: Locker dispatching seq 698423 type 0 context 11
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumSendUpdate: Dispatching seq 698423 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumSendUpdate: Locker updating seq 698423 type 0 context 11
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumSendUpdate: completed update seq 698423 type 0 context 11
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] Thread 0xbf8 UpdateLock wait on Type 0
b50:bf8.04/21[03:43:35.616](698423) INFO [GUM] GumpDoLockingUpdate: lock was free, granted to 2
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumpDoLockingUpdate successful, Sequence=698424 Generation=0
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumSendUpdate: Locker dispatching seq 698424 type 0 context 11
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumSendUpdate: Dispatching seq 698424 type 0 context 11 to node 1
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumSendUpdate: Locker updating seq 698424 type 0 context 11
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumpDoUnlockingUpdate releasing lock ownership
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumSendUpdate: completed update seq 698424 type 0 context 11
b50:bf8.04/21[03:43:35.616](698424) INFO [GUM] GumSendUpdate: Locker waiting type 0 context 11