CDR Tickets

Issue Number 4023
Summary Maintain PDQ Partner List on CDR Server
Created 2016-02-09 14:18:53
Issue Type Improvement
Submitted By Englisch, Volker (NIH/NCI) [C]
Assigned To Englisch, Volker (NIH/NCI) [C]
Status Closed
Resolved 2016-04-22 16:23:09
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.178578
Description

Every week after our Friday publishing job finished our PDQ partners are being notified indicating that the updated data is available. This notification is submitted from the FTP server that's using a file with all of the partner email addresses. This file needs to be updated nearly every week which requires a CBIIT ticket.
We want to implement a process that allows us to maintain the address file in the CDR and eliminates the need of CBIIT.

Comment entered 2016-02-18 12:03:43 by Learn, Blair (NIH/NCI) [C]

Estimate: 8 points

Comment entered 2016-04-21 20:50:57 by Englisch, Volker (NIH/NCI) [C]

The following programs have been copied to SVN:

  • R13945: get-pdq-contacts.py

  • R13945: update-pdq-contact.py

The Linux scripts have not yet been versioned:

  • get_stats

  • SendEmail.py

Comment entered 2016-04-22 16:16:44 by Englisch, Volker (NIH/NCI) [C]

The program SendEmail.py which is run as a cronjob in order to notify the PDQ partners has been modified to select the partner data from the CDR instead of reading a locally stored file. Bob created a loader program that reads the flat file named ftp_vendors.db and populates the tables

  • data_partner_product

  • data_partner_org

  • data_partner_contact
    Additionally, Bob created a view combining the information which mimics the original flat file

  • pdq_contact

Once a data partner has been notified the corresponding record will be updated with the notification count increased by 1 and the notification date will be updated.

In the event of an email failure while the program is running a new option has been created: --exclude-after
The exclude-after option requires a date/time stamp. Any partner notified after the specified time will not be notified again. Only partners for which the notification date is before the specified time will be notified.
The format of the time stamp provided for the exclude-after option is YYYY-MM-DD_HH:mm:SS where the time specified has to be in a 24 hour format.
A notification rerun would be submitted with the following command:

./SendEmail.py --livemode --product=CDR --exclude-after=2016-02-01_13:00:00
Comment entered 2016-04-22 16:22:51 by Englisch, Volker (NIH/NCI) [C]

The following programs have been copied to subversion:

  • R13954: SendEmail.py

  • R13957: get_stats.py

The following test scripts have been created:

  • R13953: update-data.py

  • R13953: ask-for-data.py

Comment entered 2016-05-18 17:18:54 by Englisch, Volker (NIH/NCI) [C]

The documentation for this command has been updated.

Comment entered 2016-05-25 12:37:43 by Englisch, Volker (NIH/NCI) [C]

The new system works perfectly - as long as we don't enter partner names including unicode characters. A bug ticket has been entered to fix the unicode error.

Closing ticket.

Elapsed: 0:00:00.001829