CDR Tickets

Bryan says he wants this to be included as part of Feynman.

Here's the high-level view of what will be done for this task:

1. rewrite /Production/prod/bin/SendEmail.py as a CDR Scheduler task

2. rewrite /Production/prod/bin/PDQAccessReport.py as a CDR Scheduler task

3. replace /Publishing/FtpExportData.py with script re-implementing functionality in the remaining /Production/prod/bin scripts

4. move /Production/prod/docs to /Publishing/docs in svn and eliminate /Production from repo

5. drop /Publishing/FtpOtherData.py (and remove invocation from publishing_task.py)

6. get CBIIT to provide Windows shares for the directories managed by the sFTP server

7. turn off the Linux cron job and test

8. get CBIIT to provide archival backup copy of nciws-p195-v:/home/cdroperator

This summary omits many details, but I think it includes all the major categories of work that needs to be performed. Have I left anything out, ~volker?

I think you covered it all. One thing we may want to do is to tar up all of the logs on the linux server (under .../vendor/log) or maybe the last 3 years or so. However, there is only limited value to those log files but one never knows if someone needs to know if organization X had ever been a partner.

Not a bad idea. Added to the list.

The Jobmaster has been modified to use sftp-export-data.py instead of FtpExportData.py for creating the tar files and updating the PDQ partner data and the program sftp-export-data.py is using rsync to update the data files on the FTP (test) server.

Currently running a full (weekly) test job on DEV.

The weekly publishing job ran successfully on DEV. Currently running a test publishing job on QA.

Steps to perform as part of the roll-out to STAGE/PROD:

1. create directory D:\cdr\sftp_shadow

2. populate D:\cdr\sftp_shadow with content from FTP server

3. copy new D:\cdr\publishing\sftp-export-data.py

4. copy updated D:\cdr\Scheduler\tasks\publishing-task.py

5. copy updated D:\etc\cdrapphosts.rc

6. copy ssh-key (pub, private) D:\etc\cdroperator_rsa

7. set ssh-key permissions

8. run fix-permissions on the updated files

9. restart scheduler service

I created a script that will allow CBIIT to perform the above steps.
After the step is completed, CBIIT will have to manually connect to the SFTP server via SSH using the account running the Jobmaster.py job in order to update the RSA key fingerprint for the host.

~bkline, are we going to submit a ticket to have these changes setup on STAGE? Since we are currently using a test server that John Ribeiro had setup I would think it's of no use to have these changes on STAGE until the sFTP server has been setup to be used from all tiers, right?

Agreed.

Going through the crontab file on DEV there are two jobs we haven't decided yet how to handle.

• PDQAccessReport.py, This job goes through the log file of the previous month to identify those PDQ partners who have downloaded the PDQ data.

• Copy logs; a command to copy and save the latest monthly log file locally. CBIIT keeps a limited number of logs but we like to keep more. We will need to decide where to store these logs in the future and those currently stored on the DEV server.

I don't believe CBIIT would want us to keep a copy of the log files locally under the cdroperator account, so we will need to discuss where on our CDR server we would want to keep these and then adjust the PDQ Access Report accordingly.

It looks like the questions we want to ask are:

1. Will it be possible for the account we will be using for posting data for the PDQ partners to also fetch the logs we need?

2. If so, does CBIIT want to restrict the portions of those logs we pull down?

3. If the answer to the first question is "no" then can we get CBIIT to set up a cron job which will provide us with the excerpts we need?

Why don't you go ahead and ask John these questions, and ask him if he needs us to open a separate ticket for this activity.

If so, does CBIIT want to restrict the portions of those logs we pull down?

CBIIT is already doing that. We're only allowed to view a subset of the full FTP log file but we may need to copy those logs to the CDR server in order to run the PDQAccess report.

I'll check with John how he wants to handle this.

John doesn't want us to run any application-specific jobs on that server, so we should probably modify our PDQAccess report to retrieve the monthly log files and create the report with the downloaded data.

Done. I ran one more test from the user interface. Let me know if it gets through safely. I added the extra documentation in the source code that we decided on during today's code walkthrough. Please review the following section (from the latest version checked into SVN):

• lines 131 and following

• lines 153 and following

• lines 339 and following

Thanks.

~volker: I'll be doing some work over the next couple of days to identify what needs to go into the patch for this ticket. I'm assuming that the branch is reasonably stable at this point, at least in terms of which files have been changed, added, dropped, etc. (I'll be able to pick up last-minute changes for this set of files). If you end up needing to add, modify, or delete a file which hasn't been affected so far in the branch, please let me know so I can adjust my deployment package accordingly.

Thanks.

I'll take a quick look but I'm pretty certain there won't be any additions - deletions could be delayed if needed.

For example, I see these in d:\Inetpub\wwwroot\cgi-bin\cdr on DEV

ResetMaxJobId.py
ResetNextJobId.py
SetNextJobId.py

... but not in version control. Are these perhaps things you had been intending to include in the patch? Or are they for further in the future?

These files are part of OCECDR-4215 which is listed under Hawking. Since there isn't a branch for Hawking and we're waiting to finish up Feynman I don't have these changes versioned yet.

I've removed ResetMaxJobId.py and ResetNextJobId.py since these two are not needed anymore.

I've updated the records that were coming up with blank organization names.
The report looks good.

Sounds good, thanks.

I have updated cdrapphosts.rc (svn and dev), using ncias-p584. If you prefer, you can replace it with "cancerinfo" which should work just as well.

Nope! This is the place where I find the server names CBIIT understands. I can remember cancerinfo but I have a hard time with ncias-p584. :-)
I prefer to keep it unless we want to add another row with the label SFTPC (for CNAME). I might do that. We have done the same for APP, GLOSSIFIER, and EMAILERS.

I see you created a new label in cdrapphosts.rc: PUBFTP, instead of using SFTP. I realize that SFTP didn't really specify the FTP server but the application server to copy the data to its final location.
Wouldn't it make sense to use SFTP for the entry for the FTP server now?

I could be wrong, but I think you added that label yourself with commit r14849. I don't have any strong feeling one way or another what you call it. In fact, I almost put the server name in the code when I was mocking things up in my sandbox early on, since (a) the server's name (that is, the one we'll use when we go live, not John's testbed) is already public; and (b) in this case, it's not the host name which changes for each tier, but the path on the same host (so we don't have the same incentive for putting this host name in the cdrapphosts.rc file as we do for the machines which change name). I do agree that even if you were to adopt that approach, it would be good to have the CBIIT internal server CNAME in the file as a memory aid.

On a related note, Publishing/Jobmaster.py should be removed, right?

You're probably right (subversion wouldn't lie, would it?).
I probably wasn't sure if there was a need to have both servers available. I'll go ahead and use SFTP, create SFTPC and update sftp-export-data.py.

I've made those changes (R14897).

subversion wouldn't lie, would it?

Not as easily as git can, anyway. :-)

One last question: these aren't in SVN but they're in the file system on DEV:

• cdr/ClientFiles/Display/PublishingSystem.css

• cdr/ClientFiles/Display/PublishingSystem_structure.css

Are these intended for a later release, or are they part of this patch?

(Looking forward to the ability git will encourage to have branches for all our work, eliminating the problems we've run into, having to tiptoe around Subversion's crankiness with merging. Then I won't have to pester you about feature- or bug-related changes that aren't lined up for the next release.)

This is for Hawking: OCECDR-4293

You're not pestering! I'm glad you're so careful. You've noticed plenty of problems that I would have overlooked in the past and 'Yes', I agree that git will likely make things a little easier.

This ran successfully (with one anomaly) on QA this morning at around 2 am. The anomaly is that there's an extra file which I thought should have been deleted but wasn't:

drwxr-xr-x.  2 cdroperator GP-pdq        398 Aug 17 11:02 docs
drwxrwxr-x. 12 cdroperator GP-pdq        935 Aug 25 02:11 full
-rwxrwx---.  1 cdroperator GP-pdq     204800 Aug 16 12:15 full.tar <-- SHOULDN'T BE HERE???
-rwxrwx---.  1 cdroperator GP-pdq 1685812203 Aug 25 01:57 full.tar.gz

The full.tar file was put in the directory on August 16 when I performed my preliminary test of the rsync command to verify that the server was reachable by the rsync command by the cdroperator account. I expected that it would be dropped because its name matches the spec we give the rsync command for what to process ("full*"). However, I may be misinterpreting the man page. It's possible that deletion of files which have been removed from the source only applies to the contents of directories which match the spec given on the command line. This interpretation seems to be confirmed by the absence of "removed" CTGovProtocol documents in the pdq-dev pile. I have moved full.tar to full/full-test.tar under pdq-qa to confirm this theory and I'll check after the next test publishing run. That behavior would be acceptable.

I have moved full.tar to full/full-test.tar under pdq-qa to confirm this theory and I'll check after the next test publishing run.

Theory confirmed. The full/full-test.tar was removed as expected.

As far as I can tell everything is working properly except for the email notification tracked in OCECDR-4301.

Closing this ticket.