CDR Tickets

Issue Number 4164
Summary Publish Preview error
Created 2016-10-06 16:19:58
Issue Type Improvement
Submitted By Osei-Poku, William (NIH/NCI) [C]
Assigned To Englisch, Volker (NIH/NCI) [C]
Status Closed
Resolved 2016-10-10 13:57:29
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.195943
Description

There appears to be a Publish Preview error that is preventing summaries from successfully generating PP. The error message displayed is:

CDRPreview web service error: Xml data validation error,The 'Err' element is not declared.Validation error occurred when validating the instance document.,11,2

Comment entered 2016-10-06 16:42:28 by Juthe, Robin (NIH/NCI) [E]

CDR IDs of some documents generating this error include 62855 and 62863. It does not seem to be affecting all summaries.

Comment entered 2016-10-06 17:39:15 by Englisch, Volker (NIH/NCI) [C]

For those summaries that show this error are you able to run the QC report? I've noticed the one summary William gave me as an example is also failing when running the QC report and this is true even for the latest publishable version.
I don't think this is related to the documents. We could try to have CBIIT restart the CDR service but I want to see if I can find some information in the log files first.

Comment entered 2016-10-06 18:31:04 by Englisch, Volker (NIH/NCI) [C]

I'm adding Bob to this ticket since I'll be out tomorrow.

It appears to me there is a problem with the CDR server. Many of the big summaries are failing to get denormalized causing the PP and QC reports to fail. Looking in the log file CdrLogErrs I see the CdrServer is getting restarted with an error DB Write Failed. This error is being logged since the 29th at a rate of about once a minute.

Comment entered 2016-10-06 18:38:51 by Juthe, Robin (NIH/NCI) [E]

I am unable to run a QC report for either doc. I'm getting the following error message: <Errors><Err>Unexpected exception caught.</Err></Errors>

Not sure if this info is helpful, but in case it is:

1) Stacy was able to run 62855 in Publish Preview earlier this afternoon without a problem. I would guess around approximately 2:30pm. I tried running it around 3pm and received an unformatted version; when I tried to call it up again (around 3:30), I began getting the validation error message. No edits were made to the document in between these attempts.

2) Both affected summaries link to modules. (William, have others reported additional summaries with this problem, or is it just these two so far?)

Thanks!

Comment entered 2016-10-06 18:48:19 by Englisch, Volker (NIH/NCI) [C]

From the sample documents that I have seen those documents are all on the "pretty big" end of the scale. I would be interested to know if the problem can also be seen for small documents. If it's only the big documents the problem may be related to memory usage and our weekly job running tonight at 9:30pm to restart the CDR service might fix the problem.

I think it's safe to say that there is nothing wrong with the documents themselves.

Comment entered 2016-10-07 07:28:32 by Kline, Bob (NIH/NCI) [C]

CdrServer is getting restarted with an error DB Write Failed.

The DB Write condition is expected when the server is shutting down. The real error causing the restart was the bind error complaining that the socket was already in use. That smells like the problem we got when CBIIT had two service managers in play.

Comment entered 2016-10-07 07:35:34 by Osei-Poku, William (NIH/NCI) [C]

PP seems to be working for the documents that failed yesterday.

Comment entered 2016-10-10 13:56:53 by Englisch, Volker (NIH/NCI) [C]

The scheduled job to restart the CDR Service ran at 9:30pm on Thursday night. According to the logs the error messages stopped at this time and the PP report for summaries resumed work again.

Closing ticket.

Elapsed: 0:00:00.001269