CDR Tickets

BZDATETIME::2012-08-06 16:17:38
BZCOMMENTOR::Alan Meyer
BZCOMMENT::1

I'm working on this as my highest priority. Volker will restore the old CdrServer on Bach tonight. No one will be able to add a new PermaTargId on Bach until I get this fixed.

BZDATETIME::2012-08-06 19:53:52
BZCOMMENTOR::Alan Meyer
BZCOMMENT::2

I found the bug. Somehow, between the time of my last test and
the time that I last rebuilt the server, I must have dropped an
'&' character from a variable reference (computers are so picky!)
The result was that all of the link validation was disabled, not
just the duplicate cdr:id validation. More precisely, all of the
validation took place but it updated a copy of the error control
object that holds all the errors instead of the real object
itself that is used to report errors to the user and update the
database.

I'm going to restore the old, working copy of the CdrServer on
Bach. That will allow everything to work except for the
PermaTargIds. There are no active PermaTargIds on Bach at the
present time, so I don't think that will be a problem for
tomorrow's operations.

Then I'm going to look at our revalidation program and see what I
can do to find any errors that slipped through. There is a
chance that the only invalid publishable documents are the ones
actually reported by cancer.gov since anything made publishable
and actually published with a link error in it would have failed
cancer.gov validation. So although the bug was very serious, I
think the damage to the database is contained to a small number
of documents that a revalidation will find and report.

I'll ask Volker to do his testing on the fixed software on Mahler
tomorrow morning and, if it looks good, we'll promote it to Bach
before I go home tomorrow night.

I'll be leaving town for a vacation on Thursday. Hopefully
everything will be okay while I'm away. If not, the interirm
work around will be to, again, restore the oldver version of the
server (the one named "CdrServer-20120723.exe" which was created
on 03/23/2011, and replace on Bach with the buggy one on
07/23/12.)

I'll report more before I leave tonight, and I'll be back at work
tomorrow morning.

BZDATETIME::2012-08-06 20:11:03
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

(In reply to comment #2)

> ...
> I'm going to restore the old, working copy of the CdrServer on
> Bach.
> ...

Done.

BZDATETIME::2012-08-06 21:20:13
BZCOMMENTOR::Alan Meyer
BZCOMMENT::4

I revalidated all of the Summaries on Bach, performing link
validation only. These are revalidations of the current working
documents, not the publishable versions. I have attached the log
file containing error reports from 15 of the 655 summaries. I
don't know if any of these errors are unexpected.

I made a modification of the revalidation software to report if
the current working document is the same as the last publishable
version. It didn't report any of the CWDs as the same as the
publishable versions, but I'm not confident of that because I
haven't had time to thoroughly test it.

I'd like to make a revision of the program to enable it to
revalidate publishable versions, but that would take more time
than I have tonight. I'll look at it tomorrow.

Since it won't interfere with anything Bach is doing tonight (I
won't update the database, just read it), I'm going to launch a
revalidation of the entire database and let it run tonight.

Attachment SummaryReval.log has been added with description: Summary link revalidation

BZDATETIME::2012-08-07 11:18:30
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::5

(In reply to comment #2)
>> I'm going to restore the old, working copy of the CdrServer on
> Bach. That will allow everything to work except for the
> PermaTargIds.

Alan, I am wondering if it is okay now to add PermaTargIds to summaries. We do have a request to add the IDs to 4 summaries on Bach.

BZDATETIME::2012-08-07 11:24:13
BZCOMMENTOR::Alan Meyer
BZCOMMENT::6

(In reply to comment #5)
> (In reply to comment #2)
> >> I'm going to restore the old, working copy of the CdrServer on
> > Bach. That will allow everything to work except for the
> > PermaTargIds.
>
> Alan, I am wondering if it is okay now to add PermaTargIds to summaries. We do
> have a request to add the IDs to 4 summaries on Bach.

It can't be done yet. The server running on Bach does not have the PermaTargId capability in it. Volker and I will do some more testing on Mahler then, if all goes well, install the PermaTargId version on Bach this afternoon.

The system will be down for about 30 seconds when I make the switch, but it should be invisible to everyone. I'll send out an email to everyone when it's ready.

BZDATETIME::2012-08-08 00:56:24
BZCOMMENTOR::Alan Meyer
BZCOMMENT::7

I did more testing of the link validation and PermaTargId processing on Mahler and more batch validation on both Mahler and Bach. I did not discover any other bugs on Mahler and did not find any other Summary errors on Bach that I could attribute to this problem.

I installed the fixed server on Bach and tested by creating a link error in one of the two documents that had the error before. I ran validation without saving the document. The new server reported it okay. I unlocked and abandoned the document (no real XML documents were harmed in making this movie.)

I then tested on Franck, which still had the buggy server. The link error was not reported. I the replaced the server on Franck and tested again. The error was now reported.

I think we're good to go on Bach to add PermaTargIds if needed and do all of the other kinds of link validation.

If I'm wrong, Volker or Bob can restore the older, safe CdrServer, as before. See comment #2 above, however the old one won't handle PermaTargId creation.

I'll be leaving town on Thursday for a week's vacation. I'll be back on Friday, August 17, and back in the office on Tuesday, August 21.

BZDATETIME::2012-08-08 00:56:46
BZCOMMENTOR::Alan Meyer
BZCOMMENT::8

Setting status to resoved-fixed.

BZDATETIME::2012-08-30 15:36:08
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::9

(In reply to comment #8)
> Setting status to resoved-fixed.

We decided to Close this issue in today's meeting.
Closing issue.