CDR Tickets

Attachment RE Missing transfer date.htm has been added with description: Missing Transfer Trade

BZDATETIME::2010-04-05 15:21:34
BZCOMMENTOR::Volker Englisch
BZCOMMENT::1

As I have already suggested in the attached email thread the problem is not that fact that the publishing steps had to be run manually but the fact that they were run at a different time.

From what I can tell at this point this is what should and did happen:

• The CTGovTransfer job selects all InScopeProtocols from the query_term table
  for which the CTGovOwnershipTransferInfo (CTOTI) block exists and writes
  these protocols in a file to compare with the previous CTGovTransfer job
  output. If there exists a protocol in the new file that didn't exist last
  time and email is being submitted including this new file.
  This method ensured - when the program was written - that, in case we
  couldn't run the program at any given time, the next time it ran it
  picked up all missing protocols.

Note here that we are looking at the query_term table containing information
for the CWD because the document is invalid without a transfer date and
therefore a publishable version containing the CTOTI block doesn't exist
for this protocol .

• For the missing protocols, it appears, that - because the publishing run
  had failed and was run manually at a later time - another process affecting
  the results of the selection for transferred protocols had already taken
  place.
  At 6am every day the CTGovImport process ran and created new CWDs for
  the missing protocols. These protocols were now listed as CTGovProtocols in
  the query_term table and therefore were ignored by the CTGovTransfer
  selection.

At this point I need to think about a solution to prevent this from happening again.
The simplest solution might be a report that could be run any time the publishing job fails and needs to be rerun manually. I believe we already have a report like this in the ad-hoc query list called "Transfer Protocols without Transfer Date (CTGov)".
Another option could possibly be to run the CTGovImport job as part of the publishing job.
I'll discuss these and any additional options with Bob and Alan.

BZDATETIME::2010-04-28 18:36:06
BZCOMMENTOR::Volker Englisch
BZCOMMENT::2

Kim, you had said in an earlier email that you need the email of the transferred protocols. Is this still the case or would - in the event of a failed publishing run - the existing ad-hoc report 'Transfer Protocols without TransferDate (CTGov)' be sufficient?

I would add the latest update date to identify the protocols missed due to the previous publishing failure.

BZDATETIME::2010-04-28 22:27:42
BZCOMMENTOR::Kim Eckley
BZCOMMENT::3

(In reply to comment #2)
> Kim, you had said in an earlier email that you need the email of the
> transferred protocols. Is this still the case or would - in the event of a
> failed publishing run - the existing ad-hoc report 'Transfer Protocols without
> TransferDate (CTGov)' be sufficient?
> I would add the latest update date to identify the protocols missed due to the
> previous publishing failure.

Yes, it would still be helpful to have the email.

BZDATETIME::2010-05-03 11:31:09
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::4

The publishing job failed over the weekend (OCECDR-3136). This time we received the email notification but as expected, the email reported only one document (CDR0000562439), which was the only which did not convert among 11 others. It will be helpful to generate the email for these protocols also.

301622
331689
369699
393548
459744
491633
513019
538993
573125
582311
655386

BZDATETIME::2010-05-03 14:13:07
BZCOMMENTOR::Kim Eckley
BZCOMMENT::5

(In reply to comment #4)
> The publishing job failed over the weekend (OCECDR-3136). This time we received
> the email notification but as expected, the email reported only one document
> (CDR0000562439), which was the only which did not convert among 11 others.

Why was this the expected behavior? Why would one trial appear and not the others?

BZDATETIME::2010-05-03 18:50:46
BZCOMMENTOR::Volker Englisch
BZCOMMENT::6

I've created the program to submit the email in the case that the publishing job failed and the InScopeProtocols didn't get picked up before they had been converted to CTGovProtocols.

I've send the output from BACH to William for review.

BZDATETIME::2010-05-04 08:39:57
BZCOMMENTOR::Kim Eckley
BZCOMMENT::7

(In reply to comment #6)
> I've created the program to submit the email in the case that the publishing
> job failed and the InScopeProtocols didn't get picked up before they had been
> converted to CTGovProtocols.
> I've send the output from BACH to William for review.

Was there a publication failure last night?

We flagged trials for transfer last night and did not see an email last night at all. Would these trials have ended up on the list you sent to William?

Thanks!

BZDATETIME::2010-05-04 09:32:57
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::8

(In reply to comment #5)
> (In reply to comment #4)
> > The publishing job failed over the weekend (OCECDR-3136). This time we received
> > the email notification but as expected, the email reported only one document
> > (CDR0000562439), which was the only which did not convert among 11 others.
>
>
> Why was this the expected behavior? Why would one trial appear and not the
> others?

There were 11 trials in the email. From the doc history, it appears the transfer blocks were added to the records last Friday 4/30. I have forwarded the email to Kim and Ning for processing.

BZDATETIME::2010-05-04 11:17:44
BZCOMMENTOR::Volker Englisch
BZCOMMENT::9

I've rerun the program on BACH to list all protocols created since 2010-05-02.

William, please check if the email includes the missing documents.

BZDATETIME::2010-05-05 09:27:33
BZCOMMENTOR::Kim Eckley
BZCOMMENT::10

More confusion.

On last nights report, there are trials that were flagged for transfer on 5/3 and 5/4.

However, there are still a large number of trials that we haven't seen an email for that were flagged for transfer on 5/3.

Why did some show up on last nights report and others not?

BZDATETIME::2010-05-05 10:12:13
BZCOMMENTOR::Volker Englisch
BZCOMMENT::11

(In reply to comment #10)
> On last nights report, there are trials that were flagged for transfer on 5/3
> and 5/4.

I am assuming you are talking about the email that was submitted to William for review yesterday morning, right? Or are you referring to the one that was created as part of the publishing job?
Obviously, the email we're submitting the day after a publishing job failed is using different selection criteria than the report submitted as part of the publishing job. The email you are referring to is basically a modified version of the ad-hoc report 'Transfer Protocols without TransferDate (CTGov)'. This query picks up all CTGovProtocols without transfer date. In order to include only the latest protocols I am passing a date parameter. For the run from yesterday I picked 2010-05-02 as the date. I figure it's better to be more inclusive than to miss some.

> However, there are still a large number of trials that we haven't seen an
> email for that were flagged for transfer on 5/3.

I would need to get some examples. Maybe our selection criteria isn't correct yet for the query or maybe the TransferDate had already been entered?

> Why did some show up on last nights report and others not?

Please give me one or two of the protocol numbers for both categories and I'll look into it.

BZDATETIME::2010-05-05 10:20:20
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::12

(In reply to comment #9)
> I've rerun the program on BACH to list all protocols created since 2010-05-02.
>
> William, please check if the email includes the missing documents.

This email certainly has a lot of the trials on it - a total of 82. Ten of the 82 were marked for transfer on 4/30 and the remaining trials were marked for transfer on 5/03. I looked at doc history of all the 82 trials and I did not see any records marked for transfer around the time the publishing job failed on 1/29/2010 and 03/19/2010. So the current email appears to account for the most recent failures which prompted starting the publishing jobs manually, but not the earlier failures (my assumption). It may be that there were no trials marked for transfer around those times. Kim may be able to provide more details on that.
Volker, I was able to confirm (again) that publishing job failed on 3/19/2010 but not 1/29/2010 is there a quicker way for you to double-check the dates?

BZDATETIME::2010-05-05 10:31:05
BZCOMMENTOR::Kim Eckley
BZCOMMENT::13

(In reply to comment #11)
> (In reply to comment #10)
> > On last nights report, there are trials that were flagged for transfer on 5/3
> > and 5/4.
> I am assuming you are talking about the email that was submitted to William for
> review yesterday morning, right? Or are you referring to the one that was
> created as part of the publishing job?
> Obviously, the email we're submitting the day after a publishing job failed is
> using different selection criteria than the report submitted as part of the
> publishing job. The email you are referring to is basically a modified version
> of the ad-hoc report 'Transfer Protocols without TransferDate (CTGov)'. This
> query picks up all CTGovProtocols without transfer date. In order to include
> only the latest protocols I am passing a date parameter. For the run from
> yesterday I picked 2010-05-02 as the date. I figure it's better to be more
> inclusive than to miss some.

I hadn't seen the email you sent to William - I was referring to the publication job email.

> > However, there are still a large number of trials that we haven't seen an
> > email for that were flagged for transfer on 5/3.
> I would need to get some examples. Maybe our selection criteria isn't correct
> yet for the query or maybe the TransferDate had already been entered?

CDR0000375585 - flagged for transfer 5/3 - showed up on your comprehensive list.

> > Why did some show up on last nights report and others not?
> Please give me one or two of the protocol numbers for both categories and I'll
> look into it.

595388 - transfer added 5/3 - not on email from publishing job
442104 - transfer added on 5/3 - showed up on last night's pub job email

BZDATETIME::2010-05-05 11:24:12
BZCOMMENTOR::Volker Englisch
BZCOMMENT::14

(In reply to comment #13)
> I hadn't seen the email you sent to William - I was referring to the
> publication job email.

I see. But now that you've seen the list from yesterday, Kim, are there any other documents missing?

William, there is only one more protocol that is not listed on the email report from yesterday or last night. This protocol is CDR63621 and was versioned on 2010-02-17.
I'm not sure why you are looking at those old failed jobs. If there were protocols that we've missed those are resolved by now because they don't show up on this email report.
Are you expecting for me to recreate the email report for any protocols that we may have missed due to those failed publishing jobs? We certainly wouldn't be able to do this with this report since we wouldn't be able to identify the protocols with the selection criteria we're using - which is to look for documents with a missing transfer date.

William, I looked through the log files and the dates you listed are correct for failures of the publishing jobs plus, of course, Friday last week and Monday.

BZDATETIME::2010-05-05 12:00:46
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::15

(In reply to comment #14)
> (In reply to comment #13)
> may have missed due to those failed publishing jobs? We certainly wouldn't be
> able to do this with this report since we wouldn't be able to identify the
> protocols with the selection criteria we're using - which is to look for
> documents with a missing transfer date.
>
This is fine with me.

As long as all the trials that should have displayed on the report (at the time of the publishing job failure) are currently on the new/comprehensive email report, I am Okay. In reviewing the report, I had to make sure those trials that were supposed to be on the emails at the time that the publishing jobs failed (and had to be started manually), have been captured by the current/comprehensive email. That is why I paid particular attention to the dates.
> William, I looked through the log files and the dates you listed are correct
> for failures of the publishing jobs plus, of course, Friday last week and
> Monday.

Thanks!

BZDATETIME::2010-05-13 10:41:29
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::16

Volker:
It looks like I can close this issue, right? I believe the outstanding issues have been addressed and we can only test this fully if there is a publishing job failure in the future?

BZDATETIME::2010-05-13 10:58:27
BZCOMMENTOR::Volker Englisch
BZCOMMENT::17

I believe that's correct.

The goal was to being able and create the proper "Transfer Email" even in the event that publishing had failed and to pick up all missing protocols.
I believe you had confirmed that the new procedure picked up all of the originally missing documents in which case this issue can be closed.

BZDATETIME::2010-05-13 11:01:02
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::18

(In reply to comment #17)
> I believe that's correct.
>
> The goal was to being able and create the proper "Transfer Email" even in the
> event that publishing had failed and to pick up all missing protocols.
> I believe you had confirmed that the new procedure picked up all of the
> originally missing documents in which case this issue can be closed.

Marking the issue as resolved.

BZDATETIME::2010-05-13 11:01:27
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::19

(In reply to comment #18)

>
> Marking the issue as resolved.

Issue closed. Thanks!