CDR Tickets

Issue Number 3541
Summary Report for publication document counts fails on non-production server
Created 2012-09-14 13:33:49
Issue Type Bug
Submitted By Kline, Bob (NIH/NCI) [C]
Assigned To Englisch, Volker (NIH/NCI) [C]
Status Closed
Resolved 2012-11-29 10:52:32
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.107869
Description

BZISSUE::5237
BZDATETIME::2012-09-14 13:33:49
BZCREATOR::Bob Kline
BZASSIGNEE::Volker Englisch
BZQACONTACT::William Osei-Poku

The cgi-script script CountByDoctype.py sometimes fails when invoked on a server other than the production server, because the database has been refreshed from the production server with publication jobs later than the most recent job run on the non-production server, with the result that the script can't find the files in the file system. Please fix this problem by detecting the condition that the job directory does not exist and either suppress the portion of the report that depends on walking through the file system or (perhaps better) get all the document type information from the database tables (in all cases, removing the dependency on the output files completely).

Whichever approach you take, please replace the exception handling in the affected functions with the following (what's there now doesn't show everything to the user because most of the error string is enclosed in unescaped <> delimiters, and even if you view the page source to see the string you don't get as much information as you get with the replacement code below):

except Exception, e:
cdrcgi.bail("Error collecting [whatever]: %s" % e)

Comment entered 2012-09-21 15:48:54 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-09-21 15:48:54
BZCOMMENTOR::Volker Englisch
BZCOMMENT::1

I have made all of the changes that Bob requested but I still have one puzzle to solve. When I'm counting the number of protocols published with the last export job I'm getting a different number than the number of documents in the vendor output directory. These differences may be related to the protocol transfer but I'm not certain yet.

The changes have been implemented and I also changed the program to use the lxml library to parse the documents which is significantly faster.

This is ready for review on MAHLER.

Comment entered 2012-09-21 16:06:40 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2012-09-21 16:06:40
BZCOMMENTOR::Bob Kline
BZCOMMENT::2

You probably already thought of this, but just in case: make sure you don't count pub_proc_doc rows for removals.

Comment entered 2012-09-21 16:11:52 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-09-21 16:11:52
BZCOMMENTOR::Volker Englisch
BZCOMMENT::3

I actually didn't think about it but the difference is about 100 docs rather than just the 4 that were removed.

Comment entered 2012-10-02 15:45:15 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-10-02 15:45:15
BZCOMMENTOR::Volker Englisch
BZCOMMENT::4

(In reply to comment #1)
> When I'm counting the number of protocols published with the last
> export job I'm getting a different number than the number of documents in the
> vendor output directory. These differences may be related to the protocol
> transfer but I'm not certain yet.

I've confirmed that the difference is a result of the protocol transfer.
I double-checked and found that a number of protocol documents created in the ActiveProtocol directory on Friday as part of the publishing job are now listed in the CDR as CTGovprotocols. I also confirmed that the number of "missing" documents from the InScopeProtocols is identical to the excess number of documents in the CTGovProtocol directory.

Comment entered 2012-10-08 10:38:28 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-10-08 10:38:28
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::5

(In reply to comment #1)
> I have made all of the changes that Bob requested but I still have one puzzle
> to solve. When I'm counting the number of protocols published with the last
> export job I'm getting a different number than the number of documents in the
> vendor output directory. These differences may be related to the protocol
> transfer but I'm not certain yet.
>
> The changes have been implemented and I also changed the program to use the
> lxml library to parse the documents which is significantly faster.
>
> This is ready for review on MAHLER.

How do I test this on Mahler ?

Comment entered 2012-10-08 10:55:21 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-10-08 10:55:21
BZCOMMENTOR::Volker Englisch
BZCOMMENT::6

(In reply to comment #5)
> How do I test this on Mahler ?

On Franck:
Go to
Reports -> Publishing -> Published Document Count
==> You'll see that the report will fail (old code)

On Mahler
Go to
Reports -> Publishing -> Published Documents Count
==> You'll see that the report will run without error (hopefully)

Also, FRANCK has already parts of the changes implemented and you won't see the error message that Bob mentioned but a message regarding a missing directory.
That's part II of Bob's request.

Comment entered 2012-10-08 14:46:44 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-10-08 14:46:44
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::7

(In reply to comment #6)
> (In reply to comment #5)
> > How do I test this on Mahler ?
>
> On Franck:
> Go to
> Reports -> Publishing -> Published Document Count
> ==> You'll see that the report will fail (old code)
>
> On Mahler
> Go to
> Reports -> Publishing -> Published Documents Count
> ==> You'll see that the report will run without error (hopefully)
>
> Also, FRANCK has already parts of the changes implemented and you won't see the
> error message that Bob mentioned but a message regarding a missing directory.
> That's part II of Bob's request.

Verified on Franck and Mahler. Worked as expected. Please promote to Bach.

Comment entered 2012-10-09 13:50:50 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-10-09 13:50:50
BZCOMMENTOR::Volker Englisch
BZCOMMENT::8

My initial tests had indicated that the switch to the lxml module would improve the performance of the report. However, when I am testing this report on BACH now it appears that the report is timing out.
I will have to take another look and see how I could speed things up.

Comment entered 2012-11-06 19:40:51 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-06 19:40:51
BZCOMMENTOR::Volker Englisch
BZCOMMENT::9

(In reply to comment #8)
> I will have to take another look and see how I could speed things up.

It seems that the reason for the timeout is the number of protocols that need to be processed (35,000+). This will need to be reworked into a batch report.

Comment entered 2012-11-07 18:03:08 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-07 18:03:08
BZCOMMENTOR::Volker Englisch
BZCOMMENT::10

I've rewritten the report to run as a batch job. I've made changes to the following files:
CdrLongReports.py
CountByDoctype.py

I'm currently testing the report on MAHLER.

Comment entered 2012-11-08 13:53:36 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-08 13:53:36
BZCOMMENTOR::Volker Englisch
BZCOMMENT::11

This report is ready for testing on MAHLER.

Comment entered 2012-11-13 22:04:36 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-13 22:04:36
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::12

(In reply to comment #11)
> This report is ready for testing on MAHLER.

I ran the report on Mahler earlier in the day but I never got the email notification.

Comment entered 2012-11-13 23:38:46 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-13 23:38:46
BZCOMMENTOR::Volker Englisch
BZCOMMENT::13

Could it be that something changed with your SPAM filter? Maybe the mail ended up in the junk folder.

Comment entered 2012-11-14 14:12:11 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-14 14:12:11
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::14

(In reply to comment #13)
> Could it be that something changed with your SPAM filter? Maybe the mail ended
> up in the junk folder.

It is not in my junk folder. I ran it again this morning and still no email.

Comment entered 2012-11-14 14:16:19 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-14 14:16:19
BZCOMMENTOR::Volker Englisch
BZCOMMENT::15

Of course!
It's the same problem that Bonnie had yesterday. (see OCECDR-3565)
I've restarted the publishing service and your job is running now.

Comment entered 2012-11-14 15:32:47 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-14 15:32:47
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::16

The emails finally showed up. I believe we can now promote the changes to Bach.

Comment entered 2012-11-14 18:02:00 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-14 18:02:00
BZCOMMENTOR::Volker Englisch
BZCOMMENT::17

The following programs have been copied to FRANCK and BACH
CdrLongReports.py - R10801
CountByDoctype.py - R10802
GeneralReports.py - R10803

Please verify on BACH and close this bug.

Comment entered 2012-11-15 11:28:39 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-15 11:28:39
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::18

(In reply to comment #17)

>
> Please verify on BACH and close this bug.

I get "The page cannot be found" error when I run the report.

Comment entered 2012-11-15 18:00:07 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-15 18:00:07
BZCOMMENTOR::Volker Englisch
BZCOMMENT::19

Oops! There was still some debugging code in the file.

This has been fixed and copied to BACH.

Comment entered 2012-11-27 13:33:56 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-27 13:33:56
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::20

(In reply to comment #19)
> Oops! There was still some debugging code in the file.
>
> This has been fixed and copied to BACH.

I can now run the report but the program stalls in just about a minute or so with the following message in the "Last message" column:

"Caught exception: free variable 'info' referenced before assignment in enclosing scope"

Comment entered 2012-11-28 17:42:15 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2012-11-28 17:42:15
BZCOMMENTOR::Volker Englisch
BZCOMMENT::21

I am not certain why we didn't already run into this problem on MAHLER. Anyway, the report has been fixed and I ran it successfully on BACH.
Please try it again and close this bug if it's OK now.

Comment entered 2012-11-29 10:52:32 by Osei-Poku, William (NIH/NCI) [C]

BZDATETIME::2012-11-29 10:52:32
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::22

(In reply to comment #21)
> I am not certain why we didn't already run into this problem on MAHLER.
> Anyway, the report has been fixed and I ran it successfully on BACH.
> Please try it again and close this bug if it's OK now.

Verified on Bach. Closing bug. Thanks!

Elapsed: 0:00:00.001268