CDR Tickets

Seriously? A bug?😛

Just a note to mention that this global also affects all Spanish HP summaries. I mentioned in the initial post that "This is a request for a global that will affect all Health Professional summaries." without specifically mentioning the languages but it does include the Spanish HP summaries as well. The The Abstracts in the Spanish summaries are written in English and not Spanish, so the update is the same for the English summaries as well as the Spanish summaries.

Understood.

Implemented on DEV. Test mode logs from that tier are attached. Diffs can be viewed at https://cdr-dev.cancer.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2021-11-30_11-33-17. Note that many of the documents in that run have no diffs because they were already transformed by the previous run, which aborted about halfway through when it encountered a summary with a TranslationOf element whose cdr:ref attribute was missing. The filter module was enhanced by OCECDR-5073 to catch the exception raised by this condition so that the global change job could run to completion.

Please run in live mode on DEV. Thanks!

Running now. The first few documents (62675, 62680, 62687, 62698) are done.

Live run on DEV has completed.

Looks good on DEV. Please run in test mode on QA.

I tried running in test mode on QA, but I ran into the same data error which caused a filter to blow up (missing CDR ID in a TranslationOf element, see earlier comment above). I have an enhancement to the callback which supports the filter queries to catch those data problems and prevent them from aborting the global change job, but that fix hasn't been promoted to QA yet (it's in Ohm), so I'm going to have to run the job on DEV, pointing to the documents on QA. Will do that when the remote server's smart card reader support decides it's going to recognize my aabkline PIV card and lets me log into DEV (sometimes it takes dozens and dozens of tries before it succeeds).

Jira is acting very strangely, so I'm hoping this comment gets through (and our CDR meeting isn't hobbled by Jira). The global change job still failed, even running on DEV against the QA data. The problem is the TranslationOf element in CDR256625 which is missing its cdr:ref attribute to identify the original English summary. You'll need to either fix the document or modify the requirements for the job to have it skip blocked documents. This will need to happen on all the tiers on which we run this job.

I have fixed CDR256625  by linking the TranslationOf element to the original summary and made a non-publishable version.

I tried running the job again, but the CDR QA server just disappeared after processing a few of the documents. Will try again in a while.

Failed again. CDR610260 has the same problem.

It looks like we will be running into this same problem on PROD for all blocked summaries as well because we remove nearly all links before blocking a summary. Perhaps run a query that identifies all summaries with a missing TranslationOf link so we can fix them before you run the global.

Fixed. In this case, I removed the element from the summary instead of creating the link.

Here are the ones I found on QA.

Here's STAGE:

And here's PROD:

I have removed the elements from the affected summaries on QA. You may proceed with the test run on QA. Thanks!

https://cdr-dev.cancer.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2022-01-26_16-49-31

As explained earlier, the job is run from DEV, using the data on QA. So although "cdr-dev" in the host name portion appears in the URL for the test results, the test was actually run for QA. The test run took over two and a half hours.

Please run in live mode on QA. Thanks!

Done. Elapsed time: 3.5 hours.

Verified on QA. Please run in test mode on PROD. Thanks!

The job aborted about 1/5 of the way through.

2022-02-16 13:20:10.038 [ERROR] Document 256625
Traceback (most recent call last):
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 274, in run}}
{{    self.doc = self.Doc(self, doc_id)}}
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 413, in init}}
{{    self.load_versions()}}
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 439, in load_versions}}
{{    errors = cdr.valPair(*args, tier=self.job.tier)}}
{{  File "d:\cdr\lib\Python\cdr.py", line 5140, in valPair}}
{{    result = valDoc(session, docType, doc=oldDoc, tier=tier)}}
{{  File "d:\cdr\lib\Python\cdr.py", line 2397, in valDoc}}
{{    raise Exception(error)}}
Exception: ('22018', "[22018] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Conversion failed when converting the nvarchar value 'NaN' to data type int. (245) (SQLExecDirectW)")
2022-02-16 13:20:10.069 [ERROR] Stopping after 1 errors
2022-02-16 13:20:10.085 [INFO] Run completed.
{{   Docs examined    = 163}}
{{   Docs changed     = 0}}
{{   Versions changed = 417}}
{{   Could not lock   = 0}}
{{   Errors           = 1}}
{{   Time             = 0:42:27.418887}}

Did you perhaps forget to fix the broken documents on PROD?

Yes, I forgot about it. They are fixed now. Thanks! Please run in test mode again.

2022-02-17 09:47:34.106 [ERROR] Document 610260
Traceback (most recent call last):
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 274, in run}}
{{    self.doc = self.Doc(self, doc_id)}}
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 413, in init}}
{{    self.load_versions()}}
{{  File "d:\cdr\lib\Python\ModifyDocs.py", line 439, in load_versions}}
{{    errors = cdr.valPair(*args, tier=self.job.tier)}}
{{  File "d:\cdr\lib\Python\cdr.py", line 5140, in valPair}}
{{    result = valDoc(session, docType, doc=oldDoc, tier=tier)}}
{{  File "d:\cdr\lib\Python\cdr.py", line 2397, in valDoc}}
{{    raise Exception(error)}}
Exception: ('22018', "[22018] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Conversion failed when converting the nvarchar value 'NaN' to data type int. (245) (SQLExecDirectW)")
2022-02-17 09:47:34.153 [ERROR] Stopping after 1 errors
2022-02-17 09:47:34.168 [INFO] Run completed.
{{   Docs examined    = 316}}
{{   Docs changed     = 0}}
{{   Versions changed = 825}}
{{   Could not lock   = 0}}
{{   Errors           = 1}}
{{   Time             = 1:16:12.833122}}

Fixed. Please try again.

The job blew up again, this time on CDR801406. When I gave you the list of broken documents on PROD, the expectation was that you would have all of them fixed, not for me to keep running the job over and over while they got fixed one or two at a time. And I would hope that no one would be breaking any more documents after I created that list for you, but if they did those need to get fixed, too. Did you forget that I had given you a list above for each tier beyond DEV (at your request)?

Also, ...

... because we remove nearly all links before blocking a summary.

Not sure why that's necessary, but if it is, why wouldn't you remove the element which requires the link instead of removing the link but leaving the broken element (and therefore broken document) in place?

Finally, I'll renew my offer to modify the global change script to avoid processing blocked documents. If you really don't want to fix the broken blocked documents, that might be a preferable approach rather than running the job over and over, only to have it fail after babysitting it for a couple of hours.

I have now removed the element from all the summaries. It should work now. Sorry about that.

2022-02-18 10:54:46.545 [INFO] Run completed.
{{   Docs examined    = 753}}
{{   Docs changed     = 0}}
{{   Versions changed = 1509}}
{{   Could not lock   = 0}}
{{   Errors           = 0}}
{{   Time             = 2:30:00.939101}}

https://cdr-dev.cancer.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2022-02-18_08-24-45

Thanks! Quick question. If someone has a summary checked out, does the global unlock the summary to make the update? I had at least 2 users report that summaries checked to them reported an error message "Document is not locked" when they tried to save and when I looked at the audit trail for one of them, it shows that the global unlocked it at some point when the document was locked to them and probably the global was running just around the time they got the error message.  The document ID is CDR0000777841 on PROD. 

Yes, I believe that's true.

Test logs verified. Please run in live mode on PROD. Please run this global after 6 PM when users would not be actively working in summary documents.

2022-02-24 23:18:45.025 [INFO] Run completed.
{{   Docs examined    = 754}}
{{   Docs changed     = 537}}
{{   Versions changed = 1174}}
{{   Could not lock   = 3}}
{{   Errors           = 0}}
{{   Time             = 3:10:13.569292}}
Specific versions saved:
{{  new cwd = 59}}
{{  new pub = 428}}
{{  new ver = 209}}
{{  old cwd = 218}}

Log attached.

Verified on PROD. Thanks!