CDR Tickets

It looks like the last time this happened was 9/4/24.

I was able to get the report to work. No idea why it failed.

It looks like the report failed again this weekend.

I've run it by hand, and this time it succeeded. I've seen more frequent notifications from my monitoring software for failures in both the EBMS and the CDR recently. We can't say with certainty that those are related in some way to the recent GovDelivery report failures, but it's certainly possible. For the GovDelivery failures, we have basically two paths available to us (besides just ignoring the problem):

1. Get CBIIT to track down and fix the cause(s) of the failures.

2. Rewrite the software to do more of the work in our own code, less in the database server.

The disadvantages attached to the first option include:

• CBIIT is often reluctant to track down the causes of failures on their servers

• they are quick to make things up when asked questions to which they don't know the answers

• the problems might take a long time to get resolved

• the problems might never get resolved

• we have less control over the process

The disadvantages to the second approach:

• moving work that the database should be able to do onto the client is backwards from recommended best practices and the whole purpose of using a dedicated RDBMS

• we don't learn the cause(s) of the failures, and therefore we are exposed to risks of other failures besides those in this report

• increased complexity in our own code

• adding changes to be tested

• we can't change the software for this report without a release

Let's discuss our options in Thursday's meeting.

One possible option to try would be to tweak with the timing of the schedule for these reports.  We could for instance:

• Run the English report at a different time in case the current time is competing with other tasks running at that time

• Switch running the Spanish version first and English second if it's more likely that the second report succeeds.  We would still have to run the Spanish version by hand once in a while but a larger audience would receive "their" report more reliably.

• As mentioned to Bob, since a second try typically succeeds we could test for a failure of the first attempt and then rerun the report/query.