CDR Tickets

BZDATETIME::2010-03-30 15:31:55
BZCOMMENTOR::Alan Meyer
BZCOMMENT::1

I wrote the global change and ran it in test mode on Mahler.

The elements to be changed are not indexed, but there were not
that many documents to process so I didn't bother re-indexing the
data or parsing the documents to pre-select just those that
need changing. I just processed all of them. The ones that
don't have the elements that need changing won't change. When
running in live mode they will not be written back to the
database.

Results of the test are in the global change test results on
Mahler in:

http://mahler.nci.nih.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2010-03-30_15-02-27

Many documents had no TermEnd... element and so did not change at
all. I believe that only one document, 369913, had a TermEndDate
that was not included in a BoardMembershipDetails block. That
element was unchanged by the program.

When someone has checked these, I can run in real mode on
Mahler.

BZDATETIME::2010-04-05 15:19:48
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::2

(In reply to comment #1)

> When someone has checked these, I can run in real mode on
> Mahler.

I looked at the test results and they seem fine. Please run in live mode on Mahler so that I can compare with the actual CDR documents.

BZDATETIME::2010-04-05 16:00:30
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

(In reply to comment #2)

> I looked at the test results and they seem fine. Please run in live mode on
> Mahler so that I can compare with the actual CDR documents.

I ran the global in live mode. As I was watching, I noticed
error messages scrolling past complaining about an invalid
date "0000-00-00". I halted the program to see if I was
doing something wrong, but saw that these invalid dates were
already in the data. I presume that they are placeholders
that were intentionally put there. So I restarted the program
from the beginning.

The log file has a "— Restarted ..." line to show where that
occurred. I don't think it affects any of the results.

Log file is attached.

Attachment Request4921.log has been added with description: Log file for live run on Mahler

BZDATETIME::2010-04-06 13:29:37
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::4

I have looked at a good sample of the documents and the global appears to do exactly what we meant it to do. However, as Alan pointed out already, users entered invalid dates for a lot of the records. Also, users typed in text in places where they were only to select a menu item. There are cases where the target link to the person documents no longer exists. I am not sure if it is only on Mahler. All of these issues will make a lot of the documents invalid. Even though they will not be published, I think it will be good to have them as valid documents.

I am having the same problem I had the other day of not being able to post some comments. I am trying this short one and will add the remaining comments in the next post

BZDATETIME::2010-04-06 13:30:06
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::5

>
> **I am having the same problem I had the other day of not being able to post
> some comments. I am trying this short one and will add the remaining comments
> in the next post**

Since the date issue is the one that affects many of the documents, it will be good if we can include a 'fix' in this global, assuming we do not have the actual dates for these days in a file somewhere. I think the affected dates are:

InvitationDate and
TermStartDate

On the other hand, if we cannot include it in the global, then users will have to go in later to fix the errors after the global.

BZDATETIME::2010-04-06 13:46:43
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::6

The 0000-00-00 dates were added to records for Board members who started on the Boards a long time ago and we didn't have invitation dates or actual start dates for them. So these dates can be skipped. I think the bigger question about dates is that when we changed the name of the element we also clarified the meaning of the date that goes in it--it hadn't been used consistently. The Board managers are going to have to go into each of their current member records and make sure it is being used correctly.
In terms of the other things you noted William, I also think we shoudl clean up the records and make them valid. There were a lot of gaps in training on these records but I think now everyone is clear on how to enter each field.

BZDATETIME::2010-04-06 15:04:49
BZCOMMENTOR::Alan Meyer
BZCOMMENT::7

(In reply to comment #6)
> The 0000-00-00 dates were added to records for Board members
> who started on the Boards a long time ago and we didn't have
> invitation dates or actual start dates for them. So these
> dates can be skipped. ...

If we want to leave an arbitrary date placeholder in these
elements to indicate that we don't know the date, we can make
them valid by changing them to an arbitrary value that is a valid
date string, for example the following date string will pass
validation.

0001-01-01

Another approach for the TermStartDate is to simply delete the
element where we don't have a real date. It's an optional
element. However InvitationDate is required. We'd either have
to change the schema to make it optional, or continue to use an
arbitrary date like 0001-01-01.

If we want to modify the dates changing zeroes to ones, I can
do it in this global or we can run a new one.

BZDATETIME::2010-04-07 09:52:15
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::8

For some reason I think we needed to have those dates in there, possibly for the mailer reports? Anyway, changing them to the valid 0001-01-01 date is fine with me.

BZDATETIME::2010-04-08 17:56:53
BZCOMMENTOR::Alan Meyer
BZCOMMENT::9

At our meeting today we decided to incorporate a change from '0000-00-00'
to '0001-01-01' in this global change.

I'll work on that, re-test on Mahler, and report back.

BZDATETIME::2010-04-09 00:00:19
BZCOMMENTOR::Alan Meyer
BZCOMMENT::10

I've run the program in test mode on Mahler. The test results
are not 100% useful because all of the documents have already
been transformed once in a previous live mode test. So all we
can see now is the new change that converts 000-00-00 to
001-01-01. The TermEndDate and TermEndReason changes have
already been made.

I also ran in test mode on Bach. I would have preferred to use
Franck, but the 4721 global was still running there and I didn't
want the log file to contain mixed outputs.

I discovered that there are four records that had TermEndDate
values of '0000-00-00'. I didn't expect that. Our options are:

Modify these in the global.

Modify them by hand.

Delete the element (either by hand or program) if this is an
error.

The documents on Bach are:

CDR0000369796
CDR0000369827
CDR0000369882
CDR0000369908

I have not yet installed the new ModifyDocs.py module on Bach. I
want to go through full tests of at least these two global
changes on Franck before we promote that module to Bach.

I'll start a Frank test from home tonight or tomorrow.

BZDATETIME::2010-04-09 10:39:22
BZCOMMENTOR::Alan Meyer
BZCOMMENT::11

The test mode run on Franck is complete. It does not include
any modification to handle 0000-00-00 in the TermEndDate /
TerminationDate element, though it does transform it in the
others.

If possible, I'd like for someone to give me a go ahead to
do a live run on Franck before we promote the 4721 global
to Bach. That will give us another test of the modified
ModifyDocs (global change "harness") module before we promote
it to Bach.

BZDATETIME::2010-04-09 12:01:09
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::12

(In reply to comment #11)
> The test mode run on Franck is complete. It does not include
> any modification to handle 0000-00-00 in the TermEndDate /
> TerminationDate element, though it does transform it in the
> others.
>
> If possible, I'd like for someone to give me a go ahead to
> do a live run on Franck before we promote the 4721 global
> to Bach. That will give us another test of the modified
> ModifyDocs (global change "harness") module before we promote
> it to Bach.

I looked at the test results and they look fine to me. Please proceed to run in live mode on Franck.

BZDATETIME::2010-04-09 12:40:48
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::13

I think we can modify that one document manually so it is also fine with me if you run it in live mode on Franck.

BZDATETIME::2010-04-09 13:05:12
BZCOMMENTOR::Alan Meyer
BZCOMMENT::14

(In reply to comment #13)
> I think we can modify that one document manually so it is also fine with me if
> you run it in live mode on Franck.

It's four docs, but still shouldn't take long to do.

I'll start the run now.

BZDATETIME::2010-04-09 13:12:34
BZCOMMENTOR::Alan Meyer
BZCOMMENT::15

The live run is complete. Log file is attached.

Attachment 4791.log has been added with description: Log file for live run on Franck

BZDATETIME::2010-04-12 17:11:34
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::16

Just taking a quick look at the log file, I looked at this warning:
Warning for CDR0000369740: Invalid value: 'Deceased' in element TerminationReason

and when I looked at the actual record, I couldn't find any Deceased in the record. This the record for Pappo, which shows up second in the log file as having been changed appropriately. I wonder what I am missing?

BZDATETIME::2010-04-12 20:00:13
BZCOMMENTOR::Alan Meyer
BZCOMMENT::17

(In reply to comment #16)
> Just taking a quick look at the log file, I looked at this warning:
> Warning for CDR0000369740: Invalid value: 'Deceased' in element
> TerminationReason
>
> and when I looked at the actual record, I couldn't find any Deceased in the
> record. This the record for Pappo, which shows up second in the log file as
> having been changed appropriately. I wonder what I am missing?

Looking at the log file for the run on Franck I see:

2010-04-09 13:06:01: saveDoc(369740, ver='Y' pub='N' val='N' old cwd)
2010-04-09 13:06:02: saveDoc(369740, ver='Y' pub='N' val='Y' new ver)

No warning listed.

Later I see:

2010-04-09 13:06:12: saveDoc(369759, ver='Y' pub='N' val='Y' new ver)
2010-04-09 13:06:12: Warning for CDR0000369759: Invalid value: 'Deceased'
in element TerminationReason
2010-04-09 13:06:12: saveDoc(369759, ver='N' pub='N' val='Y' new cwd)
2010-04-09 13:06:12: Warning for CDR0000369759: Invalid value: 'Resigned
(Deceased)' in element TerminationReason

Is it possible you wrote down the wrong CDR ID and really meant to
look at 369759? I checked the last version and new CWD for that
doc ID, and both have the errors that the warning complains about.

BZDATETIME::2010-04-13 08:55:37
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::18

No, I copied and pasted the information from the log file, including the CDR ID.

BZDATETIME::2010-04-13 09:00:08
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::19

The very bizarre thing is that now the report isn't showing that. It definitely was in the report yesterday, because I chose that record to look at based on the warning. And like I said, I copied and pasted the warning directly from the report. Anyway, if it isn't a problem, then let's move on.

BZDATETIME::2010-04-13 10:26:05
BZCOMMENTOR::Alan Meyer
BZCOMMENT::20

(In reply to comment #19)
> ... then let's move on.

Should I run the global live on Bach?

BZDATETIME::2010-04-13 10:31:48
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::21

(In reply to comment #20)
> (In reply to comment #19)
> > ... then let's move on.
> Should I run the global live on Bach?

Let me take a look at some of the documents on Franck (and the log), right now. Will let you know. Thanks!

BZDATETIME::2010-04-13 11:01:16
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::22

(In reply to comment #21)
> (In reply to comment #20)
> > (In reply to comment #19)
> > > ... then let's move on.
> > Should I run the global live on Bach?
> Let me take a look at some of the documents on Franck (and the log), right now.
> Will let you know. Thanks!

Please proceed to run in live mode on Bach. I already fixed the date problems with the 4 documents you provided in comment #10, on Bach.

BZDATETIME::2010-04-13 11:11:54
BZCOMMENTOR::Alan Meyer
BZCOMMENT::23

(In reply to comment #22)
...
> Please proceed to run in live mode on Bach. I already fixed the date problems
> with the 4 documents you provided in comment #10, on Bach.

Excellent. I'll run in live mode tonight.

BZDATETIME::2010-04-13 23:59:20
BZCOMMENTOR::Alan Meyer
BZCOMMENT::24

I've run it on Bach.

The log file is attached.

Attachment 4791.log has been added with description: Log file for live run on Bach

BZDATETIME::2010-04-14 16:34:03
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::25

I fixed all the warnings/validation errors in the attached log except the one below. The Target link has been deleted from the CDR.
2010-04-13 23:52:35: Warning for CDR0000369894: Target link is to deleted document.
For the ones that I fixed the data errors, I had to make a determination for some of them:
For:
CDR0000369766: - users have typed in 'New job with more responsibilities' in TerminationReason element, I selected "Resigned" for this record
CDR0000369789 - users typed in 'Retired' in the ResponseToInvitation element. I selected "Accepted" due to the differences between their InvitationDate and Termination date.
CDR0000369856 - Same as above
If the options I selected are OK then there is nothing else to do to those records.
I will be looking at some of the records the global changed and post anything I find out.

BZDATETIME::2010-04-14 16:51:20
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::26

I think the Board Managers are going to need to review all of their Board member records. I will point these out to make sure that the changes are correct.

BZDATETIME::2010-04-15 10:21:06
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::27

(In reply to comment #25)

> I will be looking at some of the records the global changed and post anything I
> find out.

I looked at some of the documents that were changed as a result of the global changes and it appears the summary did exactly what we expected. At this point, I believe it is OK to close this issue.

BZDATETIME::2010-04-15 10:34:34
BZCOMMENTOR::Alan Meyer
BZCOMMENT::28

(In reply to comment #27)
...
> I looked at some of the documents that were changed as a result of the global
> changes and it appears the summary did exactly what we expected. At this point,
> I believe it is OK to close this issue.

Sounds right to me. I'm done.

BZDATETIME::2010-04-19 11:18:18
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::29

I think it looks fine. I will ask the Board Managers to take a look at each of their Board member records but I am closing the issue now.