| Issue Number | 3587 |
|---|---|
| Summary | Restart CDR Service |
| Created | 2013-03-04 10:18:46 |
| Issue Type | Improvement |
| Submitted By | Juthe, Robin (NIH/NCI) [E] |
| Assigned To | Englisch, Volker (NIH/NCI) [C] |
| Status | Closed |
| Resolved | 2013-12-11 13:34:03 |
| Resolution | Fixed |
| Path | /home/bkline/backups/jira/ocecdr/issue.107915 |
BZISSUE::5286
BZDATETIME::2013-03-04 10:18:46
BZCREATOR::Robin Juthe
BZASSIGNEE::Volker Englisch
BZQACONTACT::Margaret Beckwith
The CDR seems to be running out of memory every 2 weeks or so. This issue is meant to log activity related to restarting the CDR service. I propose that we do this once each week until a permanent solution is in place. We can discuss when we be a good time of week/day to do this on a regular basis.
Volker, please revise the component if necessary. Thanks!
BZDATETIME::2013-06-12 13:19:34
BZCOMMENTOR::Volker Englisch
BZCOMMENT::1
I am monitoring the memory use of the CDRServer for a while now and
we do know that we have to restart the server occasionally on
BACH.
However, looking at the memory usage on the CBIIT servers it doesn't
appear that this tool will be necessary. While the memory increases on
BACH from around 8K to 450K within about 2 weeks, it has been stable at
around 15K on CBIIT-DEV for the past week. (I don't have enough data yet
for CBIIT-QA)
Do we want to install this restart tool on the CBIIT side regardless or should we just drop this for now?
BZDATETIME::2013-06-12 13:22:40
BZCOMMENTOR::Bob Kline
BZCOMMENT::2
Let's hold off for now.
The CDR server was last restarted on PROD at Aug. 29th. It's been
almost two weeks since then and we should check with Rob what the
current memory usage is.
We may not need to do anything else.
Alan, this is the output provided by Rob.
MAHLER:C:\Users\aachenr2>tasklist | grep CdrServer.exe
CdrServer.exe 4296 Services 0 207,148 K
In the OCE world we started having problems when the memory usage reached 450K.
Can this ticket be closed? If not, "Blocker" hardly seems like the appropriate status level, since we're pretty much ignoring the issue.
Before we could take a look at the memory usage we had to wait until
the CDR server stopped crashing every half an hour. I had asked Rob to
check out the usage on the 11th and Alan and I decided to give it a
little more time before we would decide if it could be closed or
not.
Since then, however, the server restarted again.
It would be OK with me to close this and re-open at a later time if it becomes an issue again. By the way, there are other issues I'm ignoring. This isn't one of them. :-)
My "ignoring" comment was in the context of the "Blocker" status. If this really were a blocker, we'd be working on it around the clock until it were fixed. :-)
I setup the scheduled job 'Restart CDR Service' on DEV:
R12161: CDRServiceRestart.cmd
This job will run every Thursday at 8pm.
It's currently setup under my own user account and I will have to have CBIIT modify it to run under a system account.
I've also set it up on QA.
I've setup a scheduled job on DEV and QA.
This ran successfully on DEV and QA. We can include this as part of our next release.
The scheduled job has been setup on PROD.
Keeping this open until after the first successful run scheduled for Thursday @ 9:30pm.
The scheduled job ran successfully last night but it seems there is no need for it because the server is restarting often enough even without it to make it unlikely we would ever "see" the memory increase too much. Since September the server restarted 49 times, 29 times in December and 14 times in January.
2013-09-13 15:28:18.607 CdrServer: Stopping
2013-10-01 09:56:55.631 CdrServer: Stopping
2013-11-14 20:10:18.088 CdrServer: Stopping
2013-11-14 20:37:13.257 CdrServer: Stopping
2013-11-14 20:38:45.315 CdrServer: Stopping
2013-11-16 17:51:53.755 CdrServer: Stopping
2013-12-13 12:13:45.193 CdrServer: Stopping
2013-12-13 12:16:48.329 CdrServer: Stopping
2013-12-13 12:50:04.330 CdrServer: Stopping
2013-12-13 12:52:49.011 CdrServer: Stopping
2013-12-13 13:04:10.560 CdrServer: Stopping
2013-12-13 13:05:36.679 CdrServer: Stopping
2013-12-13 13:11:24.592 CdrServer: Stopping
2013-12-13 13:15:17.729 CdrServer: Stopping
2013-12-13 13:16:34.108 CdrServer: Stopping
2013-12-13 13:16:52.740 CdrServer: Stopping
2013-12-13 13:21:32.727 CdrServer: Stopping
2013-12-13 13:21:57.748 CdrServer: Stopping
2013-12-13 13:25:56.652 CdrServer: Stopping
2013-12-13 13:27:55.560 CdrServer: Stopping
2013-12-13 13:32:01.817 CdrServer: Stopping
2013-12-13 13:33:21.776 CdrServer: Stopping
2013-12-14 19:32:05.314 CdrServer: Stopping
2013-12-14 20:02:50.303 CdrServer: Stopping
2013-12-20 15:02:08.606 CdrServer: Stopping
2013-12-20 15:03:33.574 CdrServer: Stopping
2013-12-20 15:08:03.595 CdrServer: Stopping
2013-12-23 15:19:26.112 CdrServer: Stopping
2013-12-23 15:22:51.191 CdrServer: Stopping
2013-12-23 15:25:54.676 CdrServer: Stopping
2013-12-23 15:27:42.663 CdrServer: Stopping
2013-12-23 15:35:53.606 CdrServer: Stopping
2013-12-27 14:11:34.564 CdrServer: Stopping
2013-12-27 14:25:52.498 CdrServer: Stopping
2013-12-27 20:01:16.981 CdrServer: Stopping
2014-01-11 13:18:10.945 CdrServer: Stopping
2014-01-13 14:42:36.046 CdrServer: Stopping
2014-01-16 16:11:19.054 CdrServer: Stopping
2014-01-23 12:29:31.886 CdrServer: Stopping
2014-01-23 12:34:31.802 CdrServer: Stopping
2014-01-23 12:44:36.809 CdrServer: Stopping
2014-01-23 13:24:31.140 CdrServer: Stopping
2014-01-23 13:44:31.666 CdrServer: Stopping
2014-01-23 14:49:31.444 CdrServer: Stopping
2014-01-23 15:19:31.876 CdrServer: Stopping
2014-01-23 15:48:43.003 CdrServer: Stopping
2014-01-28 16:13:20.716 CdrServer: Stopping
2014-01-28 16:23:48.745 CdrServer: Stopping
2014-01-30 21:30:02.722 CdrServer: Stopping
Elapsed: 0:00:00.001677