CDR Tickets

Issue Number 2885
Summary Non-Unicode messages in HTML
Created 2009-04-28 20:28:54
Issue Type Improvement
Submitted By Englisch, Volker (NIH/NCI) [C]
Assigned To Englisch, Volker (NIH/NCI) [C]
Status Closed
Resolved 2010-06-11 21:39:13
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.107213
Description

BZISSUE::4560
BZDATETIME::2009-04-28 20:28:54
BZCREATOR::Volker Englisch
BZASSIGNEE::Volker Englisch
BZQACONTACT::Bob Kline

Message from Alan
----------------------------------
The debug.log file on Bach is filling up with messages from
cdrcgi.sendPage (29969 in the last 8 months) about receiving non-
Unicode strings.

I'm sure that most or all of these are innocuous and I think we
should stop logging them - either by fixing the problems (best
solution), or stopping the logging.

Admittedly, it's not a high priority.

Comment entered 2009-04-30 22:15:43 by alan

BZDATETIME::2009-04-30 22:15:43
BZCOMMENTOR::Alan Meyer
BZCOMMENT::1

I'll knock these off one by one, from time to time.

The first one is a fix to ProtSearch.py. There were 3002
warnings in debug.log from this one. I forced the input form
into Unicode to silence the warning. Since this does not affect
the database and was a trivial change, I considered my testing
adequate and went ahead and put it into production.

I checked in the change and included a bit of debugging code that
I found in the version on Bach that had never been checked in.

Comment entered 2009-05-05 18:02:46 by alan

BZDATETIME::2009-05-05 18:02:46
BZCOMMENTOR::Alan Meyer
BZCOMMENT::2

I fixed ProtocolCitationsQcReport.py. There were 56 of
those errors.

The fix is on all three servers and in CVS.

Comment entered 2009-05-05 23:24:41 by alan

BZDATETIME::2009-05-05 23:24:41
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

Fixed AdvancedSearch.py on all three servers.

This was a hot one. There were 7158 error messages in
debug.log on Bach.

Comment entered 2009-05-06 13:30:15 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-06 13:30:15
BZCOMMENTOR::Volker Englisch
BZCOMMENT::4

I fixed the programs
ProtocolHpQcReport.py. - V1.9 (3548 occurrences)
ProtocolPatientQcReport.py - V1.5 (1946 occurrences)

The fix is on all three servers.

Comment entered 2009-05-06 13:48:33 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-06 13:48:33
BZCOMMENTOR::Volker Englisch
BZCOMMENT::5

I fixed the programs
CiteSearch.py. - V1.16 (1932 occurrences)

The fix is on all three servers.

Comment entered 2009-05-06 14:22:11 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-06 14:22:11
BZCOMMENTOR::Volker Englisch
BZCOMMENT::6

I fixed the programs
AdHocQuery.py - V1.6 ( 0 occurrences)
Filter.py - V1.32 (8327 occurrences)
Help.py - V1.4 ( 214 occurrences)

The fix is on all three servers.

Comment entered 2009-05-11 12:55:04 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-11 12:55:04
BZCOMMENTOR::Volker Englisch
BZCOMMENT::7

Here is a list of all of the programs that have been creating an error message in the debug.log since May 01, 2009 (some of these have already been fixed):
81 ActivityReport.py
2 AdHocQuery.py
4868 AdvancedSearch.py
69 BoardRoster.py
51 CTGovImportReport.py
2 CdrReport.py
1338 CiteSearch.py
2 CountByDoctype.py
11 CountrySearch.py
1 DISLists.py
2 DateLastModified.py
295 DiseaseDiagnosisTerms.py
21 DocumentsModified.py
1 EditQueryTermDefs.py
2 EmailerReview.py
6811 Filter.py
140 Help.py
2 MailerCheckinReport.py
11 ModWithoutPubVersion.py
7 NewCitations.py
1 PdqDtd.py
9 PersonLocSearch.py
9 PoliticalSubUnitSearch.py
1988 ProtSearch.py
40 ProtocolCitationsQcReport.py
2326 ProtocolHpQcReport.py
1367 ProtocolPatientQcReport.py
3787 QcReport.py
28 ShowCTGovLog.py
116 ShowGlobalChangeTestResults.py
4 help.py

Comment entered 2009-05-11 13:35:23 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-11 13:35:23
BZCOMMENTOR::Volker Englisch
BZCOMMENT::8

Correction:

> 3787 QcReport.py

54 QcReport.py
(my original grep included the numbers for ProtocolHPQcReport and ProtocolPatientQcReport)

Comment entered 2009-05-12 10:21:24 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-12 10:21:24
BZCOMMENTOR::Volker Englisch
BZCOMMENT::9

I fixed the programs
BoardRoster.py - V1.14 (69 occurrences)

The fix is on all three servers.

Comment entered 2009-05-12 17:52:08 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-12 17:52:08
BZCOMMENTOR::Volker Englisch
BZCOMMENT::10

I fixed the programs
ShowCTGovLog.py - V1.2 (69 occurrences)
ActivityReport.py - V1.8 (81 occurrences)

The fix is on all three servers.

Comment entered 2009-05-13 11:35:33 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-13 11:35:33
BZCOMMENTOR::Volker Englisch
BZCOMMENT::11

I fixed the program
DiseaseDiagnosisTerms.py - V1.2 (295 occurrences)

The fix is on all three servers.

Comment entered 2009-05-18 11:59:47 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-18 11:59:47
BZCOMMENTOR::Volker Englisch
BZCOMMENT::12

I fixed the programs
PoliticalSubUnitSearch.py - V1.3 ( 9 occurrences)
ShowGlobalChangeTestResults.py - V1.5 (116 occurrences)
CountrySearch.py - V1.4 ( 11 occurrences)
DocumentsModified.py - V1.2 ( 21 occurrences)
CTGovImportReport.py - V1.7 ( 51 occurrences)

The fix is on all three servers.

Comment entered 2009-05-28 11:24:32 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-05-28 11:24:32
BZCOMMENTOR::Volker Englisch
BZCOMMENT::13

Here is a summary of the current status of this task taking the list from Comment #7 listed by number of events:

The following programs have already been modified
OK - 6811 Filter.py
OK - 2326 ProtocolHpQcReport.py
OK - 1988 ProtSearch.py
OK - 1367 ProtocolPatientQcReport.py
OK - 4868 AdvancedSearch.py
OK - 1338 CiteSearch.py
OK - 295 DiseaseDiagnosisTerms.py
OK - 140 Help.py
OK - 116 ShowGlobalChangeTestResults.py
OK - 81 ActivityReport.py
OK - 69 BoardRoster.py
OK - 51 CTGovImportReport.py
OK - 40 ProtocolCitationsQcReport.py
OK - 28 ShowCTGovLog.py
OK - 21 DocumentsModified.py
OK - 11 CountrySearch.py
OK - 9 PoliticalSubUnitSearch.py
OK - 2 AdHocQuery.py

These programs still need to be updated but the number of events these trigger are far less then the once already changed:
54 QcReport.py
11 ModWithoutPubVersion.py
7 NewCitations.py
9 PersonLocSearch.py
4 help.py
2 DateLastModified.py
2 CdrReport.py
2 CountByDoctype.py
2 EmailerReview.py
2 MailerCheckinReport.py
1 PdqDtd.py
1 EditQueryTermDefs.py
1 DISLists.py

Comment entered 2009-08-07 13:51:15 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-08-07 13:51:15
BZCOMMENTOR::Volker Englisch
BZCOMMENT::14

I made additional changes to the following programs:
QcReports.py
EditFilter.py
PersonLocSearch.py

Comment entered 2009-10-02 15:10:59 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2009-10-02 15:10:59
BZCOMMENTOR::Volker Englisch
BZCOMMENT::15

I've checked the log files and saw that we do still receive non-Unicode string errors, which is expected since we haven't modified all of the documents yet.
However, these are occurring at a rate of maybe one or two a week on average.

If it were up to me I would probably close this issue now.

Comment entered 2009-10-29 18:48:56 by alan

BZDATETIME::2009-10-29 18:48:56
BZCOMMENTOR::Alan Meyer
BZCOMMENT::16

Volker has asked for this task since he's the one who has done
the recent work on it.

Reassigning to him.

Comment entered 2010-04-01 11:04:38 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-01 11:04:38
BZCOMMENTOR::Volker Englisch
BZCOMMENT::17

Here is a list of the programs that reported a non-unicode warning during the past 9 months (the number in parenthesis is the line number of the error):
AdHocQuery (138)
CdrReport (63)
CountByDocType (419)
DISLists (351)
DateLastModified (154)
EditQueryTermDefs (145)
EmailerReview (100)
GetGPLookupValues (72)
GetGeneticsSyndromes (58)
MenuHierarchy (235)
ModWithoutPubVersion (266)
NewCitations (206)
NewDocReport (277)
PersonLocSearch (85)
ProtocolStatusChange (321)
QcReport (1243)
SummaryCitations (260)

Comment entered 2010-06-08 15:26:39 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-06-08 15:26:39
BZCOMMENTOR::Volker Englisch
BZCOMMENT::18

(In reply to comment #13)
> These programs still need to be updated but the number of events these trigger
> are far less then the once already changed:

The following programs that we had earlier reported have been updated.
OK - QcReport.py
OK - ModWithoutPubVersion.py
OK - NewCitations.py
OK - PersonLocSearch.py
OK - DateLastModified.py
OK - CdrReport.py
OK - CountByDoctype.py
OK - EmailerReview.py
OK - MailerCheckinReport.py
OK - PdqDtd.py
OK - EditQueryTermDefs.py
OK - DISLists.py

Additionally, these programs were listed in the log files and they also have been fixed.
GetGeneticsSyndromes.py
NewDocReport.py
SummaryCitations.py
ProtocolStatusChange.py
MenuHierarchy.py
GetGPLookupValues.py
AdHocQuery.py
TermSearch.py

Comment entered 2010-06-08 15:48:19 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2010-06-08 15:48:19
BZCOMMENTOR::Bob Kline
BZCOMMENT::19

Excellent! I'll close the issue unless there are any loose ends left hanging.

Comment entered 2010-06-11 18:10:52 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-06-11 18:10:52
BZCOMMENTOR::Volker Englisch
BZCOMMENT::20

(In reply to comment #19)
> I'll close the issue unless there are any loose ends left hanging.

I think I knitted all of the loose ends together. If we find another one we should be able to fix this individually.

Comment entered 2010-06-11 21:39:13 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2010-06-11 21:39:13
BZCOMMENTOR::Bob Kline
BZCOMMENT::21

All done, thanks!

Elapsed: 0:00:00.000878