Multiple VISION processes running (NG80wvpc.exe or NGwvpc.exe) causing high CPU usage observed on NuGenesis SDMS Web servers - WKB53545

SYMPTOMS

• Over a period of days (including idle periods), Web Vision could become unresponsive to requests, because its background processes use up CPU resources on the web server.  Eventually, when enough NGwvpc or NG80wvpc processes are present using a CPU core, the system as a whole becomes unresponsive to Web VISION requests and the web server has to be restarted.
• Users are unable to log in to NuGenesis WebVision
  • The browser times out when loading the page or displays a "Page cannot be displayed" message
• Applications that make use of the Web-based NuGenesis SDMS SDK are unable to connect to SDMS
• Restarting the SDMS web server resolves the issue but it reoccurs at a later date

ENVIRONMENT

• NuGenesis 8 SR2 SDMS
• NuGenesis 9.0, 9.0.1, 9.0.2 and 9.1

CAUSE

Unknown.  The SDMS debug logs show that NGwvpc.exe gets into an infinite loop when processing a message received from the network of an abnormally large size.  The source of these large messages has not been identified.  

This was identified as a defect.  It was logged under CRI-2992 and INFLMS-20699. 

FIX or WORKAROUND

1. A software patch HF2 is available for NuGenesis SDMS 9.1.  For customers on this version, please contact Waters technical support to obtain a copy of the HF2 patch
2. The issue has been resolved in NuGenesis 9.3.  The business logic in the Web Server, responsible for managing the multiple background processes was updated to better manage the life cycle of these processes. As such, the processes no longer overload the CPU of the web server and thus Web Vision operates with more consistent performance regardless of how long it has been active.
   • For other versions of SDMS, the best workaround at this time is to implement a weekly or daily scheduled restart of the SDMS web server
3. Stop the NG80wvpc.exe / NGwvpc.exe processes using Task Manager
4. Restart the NuGenesis VISION Server service
5. The services can be restarted using a batch file, scheduled to run daily using the Windows Task Scheduler.  It is recommended to run the batch file out of office hours, perhaps around 3:00AM, when it is less likely that anyone would be logged into NuGenesis VISION.

6. The contents of the batch file would look like this:

   NuGenesis 8 (NuGenesis 9)

   NET STOP "Apache Tomcat 6"
   NET STOP "World Wide Web Publishing Service"
   NET STOP "NuGenesis VISION Server"
   TASKKILL /IM “ng80wvpc.exe” /F     (taskkill /IM “ngwvpc.exe” /F)
   TASKKILL /IM “NG80wvls.exe” /F     (taskkill /IM “NGwvls.exe” /F)
   NET START "NuGenesis VISION Server"
   NET START "World Wide Web Publishing Service"
   NET START "Apache Tomcat 6"

7. If the issue persists, enable SDMS debug logging on the machine and collect the log files.  SDMS debug logging will need to be active until the issue resurfaces, which may be more than a day.  Ensure that the installation drive on the SDMS web server - has at least 30 GB free for the log  files, and more if the installation drive is also the system drive (C:)

ADDITIONAL INFORMATION

In a normally functioning SDMS web server, there should be 1 NGwvls.exe process and 1 NGwvpc.exe process.  For NuGenesis 8, the process names are NG80wvls.exe and NG80wvpc.exe, respectively.