PDQ Issues

Issue Number	4347
Summary	Modify CG2Public Job to Process Latest Weekly Job
Created	2017-12-11 12:57:14
Issue Type	Improvement
Submitted By	Englisch, Volker (NIH/NCI) [C]
Assigned To	Kline, Bob (NIH/NCI) [C]
Status	Closed
Resolved	2018-03-17 08:58:19
Resolution	Fixed
Path	/home/bkline/backups/jira/ocecdr/issue.218135

Description

Our post-publishing job CG2Public.py copies documents from the output directory to the "Vendor" directory. It accesses the files of the latest publishing job to copy. In the rare event that another publishing job started after the weekly publishing job started but before the CG2Public job started (i.e. a hot-fix publishing job) the CG2Public job picks up the wrong directory.
We want to modify this process to identify the correct Job-ID for the weekly publishing job from the table pub_proc.

Comment entered 2018-02-16 10:45:31 by Englisch, Volker (NIH/NCI) [C]

For testing purposes on the lower tiers we would still want to be able to overwrite the default Job-ID value.

Comment entered 2018-03-14 12:00:44 by Kline, Bob (NIH/NCI) [C]

Installed on DEV. The software now determines the default job directory by picking the latest successful Export job from the pub_proc table. It is still possible to override the source directory:

$ python CG2Public.py --help
usage: CG2Public.py [-h] (--testmode | --livemode) [--export | --interim]
                    [--inputdir INPUTDIR] [--outputdir OUTPUTDIR] [--console]

optional arguments:

https://github.com/NCIOCPL/cdr-publishing/commit/47c92e3
  -h, --help            show this help message and exit
  --testmode, -t
  --livemode, -l
  --export, -f
  --interim, -i
  --inputdir INPUTDIR, -s INPUTDIR
  --outputdir OUTPUTDIR, -o OUTPUTDIR
  --console, -c

https://github.com/NCIOCPL/cdr-publishing/commit/47c92e3

Comment entered 2018-03-16 17:45:15 by Englisch, Volker (NIH/NCI) [C]

The publishing job failed. I believe the following statement in CG2Public.py needs to be modified from

query.where("pub_subset", "Export")

query.where("pub_subset = 'Export'")

Comment entered 2018-03-17 08:57:45 by Kline, Bob (NIH/NCI) [C]

Right. Same for the next condition. Fixed on DEV. Good catch.

https://github.com/NCIOCPL/cdr-publishing/commit/4778d00

Comment entered 2018-03-22 13:51:18 by Englisch, Volker (NIH/NCI) [C]

I ran into another error with this program. There were validation errors which were expected due to a DTD change causing the program to fail:

Traceback (most recent call last):
  File "D:\cdr\Publishing\CG2Public.py", line 253, in <module>
    logger.warnings("CG2Public.py - Finished with Warnings")
AttributeError: 'Logger' object has no attribute 'warnings'

I fixed the typo causing the error and was able to test the program successfully by running the Post Publishing scheduled job.

Comment entered 2018-04-10 11:54:24 by Kline, Bob (NIH/NCI) [C]

Changes are on QA.

Comment entered 2018-04-10 17:27:07 by Englisch, Volker (NIH/NCI) [C]

The program seems to be doing the right thing now.

Comment entered 2018-04-11 10:24:59 by Kline, Bob (NIH/NCI) [C]

https://github.com/NCIOCPL/cdr-publishing/commit/e910ab3

Comment entered 2018-05-15 12:18:50 by Englisch, Volker (NIH/NCI) [C]

This had been extensively tested on the lower tiers and I don't want to run multiple bogus publishing jobs on PROD in order to re-test this rare occurrence of the problem reported.
Closing ticket.

Elapsed: 0:00:00.001322

CDR Tickets