CDR Tickets

Issue Number 5240
Summary [Internal] Data Partner Expiration Message Send During Testing
Created 2023-05-15 17:37:32
Issue Type Bug
Submitted By Englisch, Volker (NIH/NCI) [C]
Assigned To Kline, Bob (NIH/NCI) [C]
Status Closed
Resolved 2023-05-31 11:14:43
Resolution Won't Fix
Path /home/bkline/backups/jira/ocecdr/issue.345904
Description

Our weekly notification to all PDQ partners only runs on PROD.  On the lower tiers, the messages that would go to a partner will swap the recipient with a developer's address to prevent spamming our partners.

During Friday's test run on QA to test modifications for Pauling it appears that one email message was actually submitted to the (former) partner instead of sending the message to the developer.  The partner Hackbright Academy received a message about the expiration of the test account from the QA server but this partner's account had already been disabled on the PROD server.  The partner was still active when the CDR database on QA got refreshed with an expiration date of 2023-04-17.  When the test job ran on QA on 2023-05-12 and the account was still listed as "active" the test job did the right thing and set the status to expired.  However, the email notification shouldn't have been submitted to the partner.

We want to ensure that in a (rare) situation like this the expiration notice doesn't get send to the partner.

Comment entered 2023-05-26 17:25:41 by Kline, Bob (NIH/NCI) [C]

I'm investigating, and I'm a little puzzled. Do we have a forwarded copy of the email message sent to the data partner from QA? Here's what the logs on QA say:

 

And here's the code that sends the email message:

 

I will keep digging but I'm having trouble figuring out how the code, which the logs clearly indicate got into the block where it recognized it's not supposed to do the PROD thing, and recorded the fact that the mail was going to be sent to you, and not to the data partner, somehow ended up sending the message to someone other than you. See why I'm confused? If the data partner really got the message, did it include the "extra" bit ("In live prod mode, would have gone to ...")? This is a little from The Twilight Zone. 😛

Comment entered 2023-05-27 08:48:08 by Kline, Bob (NIH/NCI) [C]

So here's my best theory. The software did what the Pauling code says it will do, and sent you two messages. The first message had the subject line "Expiration notice: NCI PDQ Test Account for Shalimar Lardizabal at Hackbright Academy <sdlardizabal@gmail.com> ..." and a body which started out "In live prod mode, would have gone to ...." The second message had the same subject line, with a body which started out "The following message sent to ...." You saw the second message, but missed the first one, panicked, and filed this ticket. You don't really have a message from Shalimar asking why she received an extra email notification about the expiration of the test account.

Here's my second-best theory. A hacker (possibly an alien hacker) broke into the QA server, modified the code for the notification job so that it actually sent out notifications to data partners, but only for expired accounts, leaving the logging intact (pretending that the notification was sent to you), reloaded the scheduler to force it to use the hacked code, waited until after the tests were complete, then restored the original version of the job's code, careful to preserve the original time stamp.

I don't have quite as much confidence in my fallback theory as in the first one. 😉

Comment entered 2023-05-27 08:57:35 by Kline, Bob (NIH/NCI) [C]

I guess there's a third theory. Both messages got sent to you, but the NIH mail server lost the first one (the one making it clear you were receiving the notification instead of the former data partner since the job was running on a non-production server). I can believe that theory (unless Shalimar really did send you a complaint that she received an extra notification, in which case we're back to theory #2).

Theory #4: I'm reading the code and/or the logs wrong. In that case, I'll need your help to show me what I've missed.

Comment entered 2023-05-30 08:11:08 by Kline, Bob (NIH/NCI) [C]

The partner Hackbright Academy received a message ...

Short version: what evidence do we have that they actually received a message?

Comment entered 2023-05-31 11:14:43 by Kline, Bob (NIH/NCI) [C]

Unfortunately, JIRA doesn't have Works as Designed as one of the resolutions, as other tracking systems do. As far as I can tell, the software did what it is supposed to do (and has always done): it detected that it was running on a non-production tier, and sent the notification to the developer instead of to the PDQ data partner.

Attachments
File Name Posted User
data-partner-code.png 2023-05-26 17:20:07 Kline, Bob (NIH/NCI) [C]
qa-data-partner-logs.png 2023-05-26 17:19:09 Kline, Bob (NIH/NCI) [C]

Elapsed: 0:00:00.000737