CDR Tickets

BZDATETIME::2011-04-04 13:38:52
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::1

attached file

Attachment adult_primary_liver_Apr11.txt has been added with description: adult primary liver

BZDATETIME::2011-04-04 13:39:47
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::2

attached error message

Attachment citems_errorApr4-2011.doc has been added with description: Error message

BZDATETIME::2011-04-05 23:41:01
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

I freed up some disk space and posted messages that I hope
weren't too rude to the systems administrators.

I think the first thing we need to do now is for Cynthia to go
back into the CiteMS and see if everything or anything that was
broken now works. I don't really expect it to, but I fear that
there's a lot of work ahead and I don't want to start it without
first checking to see if we could be incredibly lucky. Also test
OCECDR-3337.

Assuming it doesn't work, I'd like Cynthia to document for me
exactly what steps to take to see the problem - I presume that
would be to attempt to accept or reject one of the two indicated
citations. If I need to logon as Cynthia, let me know that.

I'd also like Cynthia or Minaxi to tell me what our deadlines are
on this. How much time do we have before the work starts
affecting other users as well and, worst of all, impacts the
editorial boards.

What I do after that will depend on how much damage there is,
when our last backup was, and how much work has been done since
the last backup.

If we have a production backup from the night of March 31 / April
1, I would like to try the following:

Restore that last backup on DEV (not production).

Run all of the imports that were run on and after April 1.
There appear to be 131 total citations.

Test to see if everything works.

If it does, then we have some reason to believe that the disk
space problems were the cause of the error. I believe that
most of the software is identical. The coversheet program is
different on dev and not up to date but I can fix that before
the test. It shouldn't affect this test anyway.

I'd also want to test to see if this, combined with the disk
space fix, also fixes OCECDR-3337. If we're lucky we might get
a twofer.

If everything works, I would then want to know how hard it is to
redo any work done between April 1 and today. What I'd be hoping
to do is restore the same backup on production, redo the few
imports, and redo whatever work was done since the failure.

If we don't have a recent backup, and/or if the amount of user
work is too great, then I'll see what I can do to fix things in
the live database. However, since I don't yet know what damage
may have been done in the database, and none of this is
documented anyway, it will be a challenge and could take a long
time.

My first step might be to take a backup of production, restore it
on test, and go to work there, so that if I make a mistake and
break something else it won't do any harm.

I think I need to go out and buy some Maalox.

BZDATETIME::2011-04-05 23:42:04
BZCOMMENTOR::Alan Meyer
BZCOMMENT::4

Oops. Posted the last comment in the wrong bug! I'll repost.

Shoulda taken the Maalox first.

BZDATETIME::2011-04-06 09:53:21
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::5

Minaxi was able to import the file that failed on Monday as well as a new file. The citations have imported correctly so again we got lucky.
We have the April citations ready to import. When do you think we can proceed with importing?

BZDATETIME::2011-04-06 09:57:35
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::6

Minaxi and I stopped work in the CMS as soon as the error occured so there is no work to restore for us. We will need to check with the board managers and Bonnie to see what work they have done.

BZDATETIME::2011-04-06 10:10:36
BZCOMMENTOR::Alan Meyer
BZCOMMENT::7

(In reply to comment #6)
> Minaxi and I stopped work in the CMS as soon as the error occured so there is
> no work to restore for us. We will need to check with the board managers and
> Bonnie to see what work they have done.

Excellent!

At this point, I would like the two of you to do any testing that you can while holding actual changes to a minimum. I've been on the phone with the chief systems administrator to try to straighten out the mess at our end.

I don't know how much more testing you need to do, but please try everything that you can think of. When that's done, if you haven't run into any other problems, then I think we can resume production.

I'll probably take my own backups of the data every night for a while and squirrel them away until I'm certain that everything is running smoothly.

I would like to offer my apologies to you and Minaxi for our poor handling of the backups and disk space issues. I'll pay closer attention to this in the future.

BZDATETIME::2011-04-06 10:32:54
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::8

We ran one more test. We published one of the citations that Minaxi imported a little while ago and everything worked fine. Without adding more citations to the db we can't really do any more testing at this point.
We have 2K citations to import and another 100+ to publish. So if we can get through all this with no errors we are good to go. Just let us know when it is safe to start importing/publishing again.

BZDATETIME::2011-04-06 10:39:57
BZCOMMENTOR::Alan Meyer
BZCOMMENT::9

(In reply to comment #8)
> We ran one more test. We published one of the citations that Minaxi imported a
> little while ago and everything worked fine. Without adding more citations to
> the db we can't really do any more testing at this point.
> We have 2K citations to import and another 100+ to publish. So if we can get
> through all this with no errors we are good to go. Just let us know when it is
> safe to start importing/publishing again.

Okay.

I'm going to take a backup of my own right now. It will take 10 minutes
or so.

When that's done, let's try importing some citations and publishing some,
checking, and if everything is okay, doing the rest and checking again.

I'll get back to you as soon as my backup is done.

BZDATETIME::2011-04-06 10:54:19
BZCOMMENTOR::Alan Meyer
BZCOMMENT::10

(In reply to comment #9)
...
> I'm going to take a backup of my own right now. It will take 10 minutes
> or so.
>
> When that's done, let's try importing some citations and publishing some,
> checking, and if everything is okay, doing the rest and checking again.
>
> I'll get back to you as soon as my backup is done.

The backup is done. Let's go ahead. If anything goes wrong, please stop
work and let me know. I'll try to monitor email most of the day.

If necessary, you can also reach me at one of these phone numbers:

410-833-5979
410-922-8646

BZDATETIME::2011-04-06 12:44:28
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::11

Minaxi has imported and published a batch of 44 and 97 citations with no errors. She is going to continue with the import of the 2K citations for this month. We will let you know how that goes.
When should we let the Board Managers and Bonnie back into the CMS to do their work? Maybe after Minaxi imports the 2K? I suggest letting Bonnie in first to do some testing before we open it up to everyone else.

BZDATETIME::2011-04-06 13:10:04
BZCOMMENTOR::Alan Meyer
BZCOMMENT::12

(In reply to comment #11)
...
> When should we let the Board Managers and Bonnie back into the CMS to do their
> work? Maybe after Minaxi imports the 2K? I suggest letting Bonnie in first to
> do some testing before we open it up to everyone else.

That sounds like a good plan to me.

BZDATETIME::2011-04-06 16:49:27
BZCOMMENTOR::Alan Meyer
BZCOMMENT::13

The silence on these issues since earlier today is wonderful. I hope it means that everybody is back in production and no problems have been encountered.

BZDATETIME::2011-04-06 16:53:43
BZCOMMENTOR::Minaxi Trivedi
BZCOMMENT::14

I have imported 1400+ citations without any problem. I will continue to import the remaining citations tomorrow.

Minaxi

BZDATETIME::2011-04-06 17:32:50
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::15

I emailed Bonnie and told her to go ahead and work in the CMS tomorrow but to let us know how everything goes.
If all goes well I think by afternoon we may be able to let everyone else continue working in the CMS.

BZDATETIME::2011-04-07 12:54:24
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::16

Minaxi has finished importing with no problems. Bonnie has been working in the CiteMS all morning and has also reported no problems. I think it is safe to open the CiteMS back up for all users.

BZDATETIME::2011-04-14 23:56:59
BZCOMMENTOR::Alan Meyer
BZCOMMENT::17

Marking this resolved-fixed. See the final comment on OCECDR-3336.

BZDATETIME::2011-04-15 11:04:09
BZCOMMENTOR::Cynthia Boggess
BZCOMMENT::18

Closing this bug. All issues resolved.