Prevent SendEmail.py from Failing
JIRA_OCECDR-3755.org

   Problem
     SendEmail.py may fail, needs to run in a loop of retries
   Research
     Looked at sendMail() in /usr/local/cdr/lib/Python/cdrutil.py
       Nothing special there.
       No retry loops
       No custom timeout setting
         Although the class constructor smtplib.SMTP() supports it
         Seems to use the default timeout = None
     What errors are we getting?
       socket.timeout?
         Raised by smtplib.SMTP if default timeout occurs
         No timeout is set
         I'm assuming this isn't the error
         Can search logs for it
           Not found in one log I've got
       Is it an exception?

   Environment
     Testing machine is nciws-d165-v-m.nci.nih.gov
       Only accessible from the DEV bastion host
       On the Bastion Host, click the putty icon
         Select ftpAndGPwb (white on black)
     Can develop in my own sandbox under ~/src
     Test as cdroperator
       sudo -u cdroperator -i
     Test directory for code is:
       /home/cdroperator/prod/bin
     Test directory for data is:
       /home/cdroperator/prod/vendor
     Log files
       /home/cdroperator/prod/log

   Changes
     Wrap the try logic on line 674
       triesLeft = 5
       delay = 0
       sent = False
       while not sent:
         try:
            ...
           # If we got here, the send is assumed successful
           sent = true
         except
           ...
           if triesLeft:
             sleep(delay)
             triesLeft -= 1
             cdrutil.log("Trying again after %d second delay" % delay)
             delay += 5
             continue
           else
             sys.exit(1) - as now
   Testing
     Preparation
       Set email addresses in
         /home/cdroperator/prod/vendor/ftp_vendors.test
         Make a backup of the existing file
         Create a new file with my test data
         Switch them back and forth
       sudo -u cdroperator -i
       That may not be enough
         May need to figure out meaning of different lines
     Test 1: Did I break anything?
       Will it send me email?
     Test 2: Will it recover from exception?
       Comment out current line 693 in notifyVendor() to find out

   Changes
     Put a wrapper around sendMail
       Use it everywhere
     sendEmail()
       Purpose
         Send email with mulitple tries
       Pass
         Same as for current sendMail
       Return
         Same value returned by cdr.sendMail
       Logic
         Define MAX_NOTIFY_TRIES
         NOTIFY_TRY_INC_SECONDS
         tries_left =
         delay =
         sent = False
         while not sent:
           try
             All as now.