CDR Tickets

Issue Number 5004
Summary Delayed email notifications from running correspondence mailers
Created 2021-07-12 15:04:20
Issue Type Bug
Submitted By Osei-Poku, William (NIH/NCI) [C]
Assigned To Kline, Bob (NIH/NCI) [C]
Status Closed
Resolved 2021-07-13 17:39:39
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.293872
Description

Robin B. reported that  there is a long delay (30-60 mins) in receiving email notifications from generating correspondence mailers on PROD. Bonnie also tried it on PROD and it appears to be the same issue as she is yet to receive the email notification. I just tried it on QA and I am yet to receive the email notification as well.

Comment entered 2021-07-12 17:30:39 by Kline, Bob (NIH/NCI) [C]

I just tested on QA and I received the email notification instantly. Please attach the email message (I need the original SMTP payload, complete with headers (not just the pasted text from the body). I can see from the logs on QA that eight seconds elapsed between when the script started handling your request and when the email message was submitted to the NIH mail server, so the problem's not in the CDR itself.

Comment entered 2021-07-12 17:51:29 by Osei-Poku, William (NIH/NCI) [C]

It looks like it is an Outlook issue. . I generated a test mailer on QA at about 2:38 PM and I just received the mailer at 5:06 PM but the email was timestamped at 2:40 PM.

Received: from SA9PR09MB4911.namprd09.prod.outlook.com (2603:10b6:806:1f::21)Received: from SA9PR09MB4911.namprd09.prod.outlook.com (2603:10b6:806:1f::21) by MN2PR09MB4988.namprd09.prod.outlook.com with HTTPS; Mon, 12 Jul 2021 21:05:38 +0000Received: from DM6PR09CA0030.namprd09.prod.outlook.com (2603:10b6:5:160::43) by SA9PR09MB4911.namprd09.prod.outlook.com (2603:10b6:806:1f::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.20; Mon, 12 Jul 2021 21:05:32 +0000Received: from DM3GCC02FT015.eop-gcc02.prod.protection.outlook.com (2a01:111:f400:7d04::208) by DM6PR09CA0030.outlook.office365.com (2603:10b6:5:160::43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.20 via Frontend Transport; Mon, 12 Jul 2021 21:05:32 +0000Authentication-Results: spf=pass (sender IP is 156.40.79.161) smtp.mailfrom=mail.nih.gov; nih.mail.onmicrosoft.com; dkim=pass (signature was verified) header.d=nih.gov;nih.mail.onmicrosoft.com; dmarc=pass action=none header.from=mail.nih.gov;Received-SPF: Pass (protection.outlook.com: domain of mail.nih.gov designates 156.40.79.161 as permitted sender) receiver=protection.outlook.com; client-ip=156.40.79.161; helo=mail.nih.gov;Received: from mail.nih.gov (156.40.79.161) by DM3GCC02FT015.mail.protection.outlook.com (10.97.8.125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4308.20 via Frontend Transport; Mon, 12 Jul 2021 21:05:32 0000Received: from nihsmtpxwayst06.hub.nih.gov (165.112.13.57) by nihexb1.nih.gov (156.40.79.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Mon, 12 Jul 2021 14:42:26 -0400DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;  d=nih.gov; i=@nih.gov; q=dns/txt; s=NIH; t=1626115347;  x=1657651347;  h=content-transfer-encoding:mime-version:from:to:subject:   message-id:date;  bh=PwOt/RKawp1uHARCBXzzTcbrBFLdY1bRmLvGxzIyyEA=;  b=KStfh21lBbT+W37/TSbFfn2ema6LAm64PrT2Y9zSUvjTHABT9VWewXZc   gxxUmDIy9qVPCM6vfUCtqNzWh1O8nhI410b6pDThejw7ob99XTVo0s2JF   3UqV0R+3a7UIm0a8GSIulds2b8T2MHomexXGgV1goShJu7FV4zy1yye0o   bBDdZ4M2ve6Qm8SDBYfsdNpIqjrhia+u4V5FoEGxDrbjxo/QRUuoTpPmu   59lMO2rxCnkgQp40EXR4a4kqi8oaO4dbYMP6D1s5JKZI+VYfdXr6+lvxy   /KQoHfJFCTjKeCPUHM0KUNFmSeCLdgBZGsYpGEk16ZdFSBkXax9GRoab   Q==;Message-Id: <2da20b$1vadjo@nihsmtpxwayst06.hub.nih.gov>Date: 12 Jul 2021 14:40:12 -0400IronPort-SDR: f40aY6QXNKuq3r+ekEjZrksdQY2V2nuxN1GhUw/i8uSwGxHVWpXjSSI8mVHpBlUH4lIREKjDTN gfpXAy/1RUOstaxwTH4VPMe54GUcff68wSwe+0PlkmCy2FDV4UnTwy8wN2Uq+olKw9yC7L/Pge i1GwG0h3xFU13eiZ9/dFcI4gLziEO9y/qRBPsNDTzoVlXNHgTT7EBklZ9iiYZDMpZaD8/rNL6m /WhpAF+gNZGYSmheTQKlOWO0xmmHrDF1fZUEmz9EddHkv54qDN3/XfTy3li+h1jiONcLIM+pyG k7w=X-SBRS-Extended: LowX-IronPortListener: non-ces-outX-IronPortListener: non-ces-outX-IPAS-Result: =?us-ascii?q?A2GZCQCPjOxg/0la54BaHgEBCxIMQIFOC4F8gREVVlIah?= =?us-ascii?q?EiJBIt+mDSBfAsBAQFGCgQBAQMEhHCCWSY0CQ4CBAEBAQEDAgMBAQcBAQYBA?= =?us-ascii?q?QEBAQEFBAICgQCFLzkNg1aBCAEBAQEBAQEBAQEBAQEBAQEBAQEWAoFeET1OA?= =?us-ascii?q?iYCSUuCEEAqAYJcD59riWeBMoEBihyBFiqHCYZgAokaK04LDAQYghOCShOCU?= =?us-ascii?q?QSEUlqBKA1ZlH2KFZ1jgy6KMZd/ARGLXZcSojCaVoIVKwoYKYM0TxmUHIkjL?= =?us-ascii?q?wI2AgYKAQEDCYwzAQE?=IronPort-HdrOrdr: A9a23:ro+5dqwP4ZuwAETNRCGhKrPwK71zdoMgy1knxilNoH1uA6qlfq WV9pkmPHDP6Qr5NEtBpTn4Ase9qBDnhPlICOsqTNKftWDd0QPCEGgF1/qA/9SJIUHDH4VmpM NdmsZFeb/N5YcTt6nHCG7RKadY/OW6X-IronPort-Anti-Spam-Filtered: trueX-IronPort-AV: E=Sophos;i="5.84,234,1620705600";    d="scan'208";a="66401912"Received: from msg-b12-ltm1_v9.hub.nih.gov (HELO nciws-q2062-v.nci.nih.gov) ([128.231.90.73])  by nihsmtpxwayst06.hub.nih.gov with ESMTP; 12 Jul 2021 14:40:12 -0400Content-Type: text/plain; charset="utf-8"Content-Transfer-Encoding: 7bitMIME-Version: 1.0From: PDQ Operator <NCIPDQoperator@mail.nih.gov>To: oseipokuw@mail.nih.govSubject: [QA] CDR Mailer Job StatusReturn-Path: NCIPDQOperator@mail.nih.govX-InternalExchange: ExInternalX-Exchange-ASBP: 1X-OrganizationHeadersPreserved: nihexb1.nih.govX-MS-Exchange-Organization-ExpirationStartTime: 12 Jul 2021 21:05:32.1703 (UTC)X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmitX-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmitX-MS-Exchange-Organization-Network-Message-Id: e0270143-1e95-47fd-363b-08d94578c916X-EOPAttributedMessage: 0X-MS-Exchange-Organization-MessageDirectionality: OriginatingX-MS-Exchange-Organization-AuthSource: nihexb1.nih.govX-MS-Exchange-Organization-AuthAs: InternalX-MS-Exchange-Organization-AuthMechanism: 10X-MS-Exchange-Organization-SCL: -1X-CrossPremisesHeadersPromoted: DM3GCC02FT015.eop-gcc02.prod.protection.outlook.comX-CrossPremisesHeadersFiltered: DM3GCC02FT015.eop-gcc02.prod.protection.outlook.comX-MS-PublicTrafficType: EmailX-OriginatorOrg: mail.nih.govX-MS-Office365-Filtering-Correlation-Id: e0270143-1e95-47fd-363b-08d94578c916X-MS-TrafficTypeDiagnostic: SA9PR09MB4911:X-MS-Exchange-AtpMessageProperties: SAX-SCL-By-Transport-Rule: ironport-filtered in EXOX-MS-Oob-TLC-OOBClassifiers: OLM:5236;X-Microsoft-Antispam: BCL:0;X-Forefront-Antispam-Report: CIP:156.40.79.161;CTRY:US;LANG:en;SCL:-1;SRV:;IPV:CAL;SFV:SKI;H:mail.nih.gov;PTR:InfoDomainNonexistent;CAT:NONE;SFS:;DIR:INB;X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2021 21:05:32.0904 (UTC)X-MS-Exchange-CrossTenant-Network-Message-Id: e0270143-1e95-47fd-363b-08d94578c916X-MS-Exchange-CrossTenant-Id: 14b77578-9773-42d5-8507-251ca2dc2b06X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=14b77578-9773-42d5-8507-251ca2dc2b06;Ip=[156.40.79.161];Helo=[mail.nih.gov]X-MS-Exchange-CrossTenant-AuthSource: nihexb1.nih.govX-MS-Exchange-CrossTenant-AuthAs: InternalX-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPremX-MS-Exchange-Transport-CrossTenantHeadersStamped: SA9PR09MB4911X-MS-Exchange-Transport-EndToEndLatency: 00:00:06.6285281X-MS-Exchange-Processed-By-BccFoldering: 15.20.4308.026X-Microsoft-Antispam-Mailbox-Delivery: ucf:0;jmr:0;auth:0;dest:I;ENG:(750129)(520011016)(706158)(944506458)(944626604);X-Microsoft-Antispam-Message-Info: v4JuJA4OSszMtD0OstMXwHYqp1F2V0MdWtGWgmZUxthlHRmUXUeul8s9BtPHwea/Q4TzEBu+3vebfx3QvAZ4YKtyf8PqkSiFoZVwyBdJtVxUJYBjC0ChSdyzSWZtGPnatWBrrxWoXGjdY41aSKnqMtQaXcjQVzuQsTVqh6rdGjek0F6YBc6p+mAKFaYgcjiHC+mgws07mbn8yPAMylJ24Xdad4MYHH9dJrkpz9qlhxhCoRCLd2BpYqoF3qDxzWluymJw6cMdvS9AD4RG4QDyo9jlUPh1JcNBfNggviuZE7CUVA73BpVDQp0lPE2hpnm3q0nUmEhjV+n3EOx2SiexXPH+6QnZfCm4cntmuJKFyyC85tcdytKN998KlFzVZu9UjEyWvFZZaS16aUNJdbEAv2p3Ql13i/prye8kbILr861loPK1rXw7NvxIVJcZNoppqz9TLMivn6eDiG1eavWFNLZzm3rz8nZzqZDxnpCZYYoCh6GDYEWLW7Gm902ol6i8zvRpRXpF5saDUl0KJ/N2a9tXjgzaXN2SmakKlZ7UokGanGUPIiogFhPgY9o9rU7mfP+DyDZc5XW26hA9kvXZ2Q==

Comment entered 2021-07-12 20:27:05 by Kline, Bob (NIH/NCI) [C]

I can report an issue to CBIIT, but only if I can get a copy of the message without the headers garbled (looks line the breaks between lines have disappeared in what you pasted into JIRA).

If you're using the Web Outlook, you need to choose View > View message details, then copy and pasted into a program which won't garble what comes from the clipboard:

Should look something like this if you're successful. Try attaching it to the issue instead of pasting it into a comment.

Comment entered 2021-07-13 09:17:04 by Osei-Poku, William (NIH/NCI) [C]

MailerOutlookIssueLog.txt

I copied and pasted it into notepad which is attached. I am also including a screenshot here.

Comment entered 2021-07-13 10:29:18 by Kline, Bob (NIH/NCI) [C]

Thanks, that's what I needed. I have filed ticket NCI-RITM0304304 with CBIIT. Will report back what they say.

Comment entered 2021-07-13 17:39:27 by Kline, Bob (NIH/NCI) [C]

From CBIIT:

Comment entered 2021-07-16 15:03:57 by Osei-Poku, William (NIH/NCI) [C]

Robin b. confirmed that the emails notifications were no longer delayed.

Attachments
File Name Posted User
image-2021-07-12-20-24-38-264.png 2021-07-12 20:24:39 Kline, Bob (NIH/NCI) [C]
image-2021-07-12-20-26-51-888.png 2021-07-12 20:26:52 Kline, Bob (NIH/NCI) [C]
image-2021-07-13-17-39-14-970.png 2021-07-13 17:39:15 Kline, Bob (NIH/NCI) [C]
MailerOutlookIssueLog.txt 2021-07-13 09:14:46 Osei-Poku, William (NIH/NCI) [C]
OutlookMailerIssue.PNG 2021-07-12 17:49:30 Osei-Poku, William (NIH/NCI) [C]
screenshotOutlookMailerIssueLog.PNG 2021-07-13 09:17:02 Osei-Poku, William (NIH/NCI) [C]

Elapsed: 0:00:00.001488