EBMS Tickets

As soon as I have the list of which messages you received, ~juther, as well as a copy of the latest notification received by JBS, I will open a ServiceNow ticket to have CIT investigate the delivery failures.

ServiceNow ticket is NCI-INC0777681 | INC8007166. ~juther, ~duganal, ~volker, and ~oseipokuw are watchers.

NIH email staff have reported back that their logs provide evidence that all five of the missing notification email messages were delivered successfully to the JBS email address, and for each message a reply was received by the receiving mail server confirming that successful delivery. It appears that the receiving mail server belongs to Mimecast, which is an email security service. It seems reasonable to conjecture, therefore, that the Mimecast service provides a front end to the JBS mail servers, screening incoming messages and detecting which should be flagged or discarded. JBS should probably contact Mimecast and have the necessary adjustments made in order to allow the travel request notifications to make it all the way to the recipient inbox. Unfortunately, the configuration of security filtering tools such as these is highly volatile, and it's reasonable to anticipate that the problem might recur.

$ dig jbsinternational.com MX


; <<>> DiG 9.10.6 <<>> jbsinternational.com MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16675
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;jbsinternational.com.      IN  MX


;; ANSWER SECTION:
jbsinternational.com.   930 IN  MX  30 mx2.dnsmadeeasy.com.
jbsinternational.com.   930 IN  MX  15 us-smtp-inbound-1.mimecast.com.
jbsinternational.com.   930 IN  MX  20 mx1.dnsmadeeasy.com.
jbsinternational.com.   930 IN  MX  15 us-smtp-inbound-2.mimecast.com.
jbsinternational.com.   930 IN  MX  40 mx3.dnsmadeeasy.com.


;; Query time: 72 msec
;; SERVER: 137.187.26.4#53(137.187.26.4)
;; WHEN: Wed Sep 27 14:31:57 EDT 2023
;; MSG SIZE  rcvd: 198 

Either I misinterpreted the original reports of the problem, or those reports were incomplete. According to Terry Ketchen (JBS) the delivery of ALL of the travel notifications (not just the reimbursement requests notifications) was failing. They have just received their first hotel request notification that they have received since mid-August, so it seems probably that their technical staff have been working with their email security vendor to resolve the problem (though we don't have any explicit confirmation from Terry—or anyone else—of any such activity).

The data which led to the conclusion that only the reimbursement request notification deliveries were failing (not the delivery of the hotel request notifications) were that JBS reported that they received the hotel request for Alan Gamis on August 28, which was ten days after the reimbursement request for Tom Gross was sent out, and which JBS said they never received.

There is one other possible explanation. The email security system which JBS uses may have introduced multiple changes to the rule set which is used to identify messages which should be considered dangerous and dropped, the first of which caused the reimbursement requests to be flagged in this way, with a later modification to the rule set expanding the set of flagged messages to also include hotel request notification messages.

I'm pretty sure this ticket can be closed.

For the historical record, this ticket was entered as a bug report, but as it turns out, the EBMS was behaving correctly. Any bugs were in the systems at JBS. 😉