CDR Tickets

Issue Number 4348
Summary Improve "Post Publishing" Process
Created 2017-12-12 14:30:45
Issue Type Improvement
Submitted By Englisch, Volker (NIH/NCI) [C]
Assigned To Kline, Bob (NIH/NCI) [C]
Status Closed
Resolved 2019-08-01 07:14:39
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.218223
Description

We identified several problems when a publishing job failed during the CG2Public step and we tried to rerun this step using the "Post Publishing" scheduled step.

Problem 1:
When the "Post Publishing" job is run it will only run successfully if no other publishing job has run since the weekly publishing job ran because the job is expecting the publishing data in the latest directory with the latest Job-ID.
Solution 1:
Identify the latest weekly publishing job ID by selecting it from the database.

Problem 2:
In "Live" mode the system expects the directory to be created in the LicenseeDocs directory not to exist. It will fail to run and update the files.
Solution 2:
Implement a step to clean out a directory created during a previous, unsuccessful run.

Problem 3:
If the post-processing job fails during the CG2Public step the step to update the FTP server will run regardless. This means that an incomplete set of documents will be copied to the FTP server. Running the step again at a later time won't create a diff report to last weekend's output but to the latest FTP update job.
Solution 3:
Create checksums of each file and store in order to create the diff report. This will allow the creation of diff reports between any publishing job, not just the current and the most recent one.

Comment entered 2018-02-15 13:53:55 by Kline, Bob (NIH/NCI) [C]

Problem #2 has been addressed in Gauss.

Comment entered 2018-06-28 14:17:26 by Kline, Bob (NIH/NCI) [C]

will take a look at problems #1 and #3.

Comment entered 2018-06-29 16:38:46 by Englisch, Volker (NIH/NCI) [C]

Problem #1 has been addressed as part of Hawking (OCECDR-4347).

Comment entered 2019-07-08 07:11:30 by Kline, Bob (NIH/NCI) [C]

I have taken care of the last problem. Can we schedule a code walkthrough for some time tomorrow, ?

https://github.com/NCIOCPL/cdr-publishing/blob/joule/Publishing/sftp-export-data.py

https://github.com/NCIOCPL/cdr-publishing/commit/adcf7d6

Comment entered 2019-07-08 15:02:51 by Englisch, Volker (NIH/NCI) [C]

Sure, maybe after lunch?

Comment entered 2019-07-08 15:48:30 by Kline, Bob (NIH/NCI) [C]

Hmm. I tried (twice) to reply with a "thumbs-up" emoji, and both times Jira not only swallowed my reply, but also suppressed the display of your reply. Yours came back with a refresh, but not mine. Another Jira bug. Anyway, after lunch is fine.

Elapsed: 0:00:00.000670