Enhanced web based management tools for the CDR

• Figure out exactly what to do
  

• Create a script or set of directions
  

• Create an issue for CBIIT
  

• Wait for CBIIT to implement it
  

• Check results, often requiring CBIIT's help
  

do this without the double efforts and delays inherent in the existing process.

• On per job basis
  

• On per account basis
  e.g. only user in group X can start job that runs under account Y

• May not be used often but,
  
  • Useful because it allows us to script changes
    

• Executable binary
  

• Python script
  

• C# program
  

• SQL script
  
  • Like sqlc or sqlq in drush
    

• Bash shell script
  

• Windows command file
  

• PHP script?
  

• Java program
  

• Probably requires that job be written for that
  

• Sends a signal to job to suspend itself
  
  • We build in checks for signals to our jobs
    

• Link multiple jobs in a chain
  

• Each job returns status information for success or failure
  

• When one completes successfully, next one starts
  

• Chains can include jobs or other chains
  
  • Detect and prevent looping?
    

• Specify that a job must not be run concurrently
  
  • Maybe that's the default
    

• Limit the number of concurrent batch jobs
  

• Ideally, a job control parameter
  

• Or we can use FileSweeper
  

• Distribution lists
  
  • Custom distribution lists for different jobs
    
  • General list for all jobs?
    
  • Add/delete addrs from list via web interface
    
  • Special lists for failed jobs?
    
    • e.g., if job fails, notify …
      
    • If job succeeds notify smaller group, or no one
      

• Might simplify some efforts
  

• Idea is:
  Start a job, e.g., in the CDR. When it's done, cause data to be sent to a Linux server. When copy is complete, start a job on the Linux server.

• Enables us to script job management
  

• Ideally
  
  • Generate a schedule control table on one tier
    
  • Export/import it to other tiers
    
  • Allow certain changes to be made globally
    
    • e.g., replace a server name, db name, etc. globally
      

• See what's running
  
  • Job name, ID, link to control info
    
  • When started, time elapsed
    
  • Parameters passed
    
  • Resources used?
    
    • CPU
      
    • Disk reads / writes
      
    • Network reads / writes
      
  • Current status
    
    • Succeeded
      
    • Failed
      
    • Running
      
    • Suspended
      
    • Waiting on dependency
      
    • Idle
      
      • Running, but nothing happening
        
    • Killed
      

• Set limits on job
  
  • If runs longer than X
    
    • Notify user
      
  • If runs longer than Y
    
    • Suspend the job
      
    • Kill the job
      

• Dependencies
  
  • Specify minimum disk (or memory?) availability
    before starting a job
    • Scheduler won't schedule job if resources are insufficient
      
    • Logs and sends email when job can't start
      
  • Resource locking
    
    • e.g., Have a way to specify that some resource must be open
      
    • Example in Windows, find out if some process has a file open
      
  • Specify other jobs that must be complete before starting this
    
  • Specify other jobs that can't be running when starting this
    

• Triggering
  Similar to dependency checking but controls what causes a job to start instead of what prevents a job from starting
  • Completion of a job starts another
    
    • Chaining is one way to do this, but not the only way
      
  • Receipt of an email
    
  • Appearance of a file or database row
    
  • Success or failure code from a script
    
  • API call from a job
    

• System records what it's doing when doing it
  

• Checks records when starting
  
  • Detects that jobs X, Y, and Z were in progress
    
  • Understands that a crash occurred
    
  • Logs info, emails info
    
  • Allows per job controls over what happens in crash recovery?
    
    • Restart job automatically?
      
    • Skip job?
      
    • Stop all processing until human input?
      
      • Alert humans, pause for instructions
        
    • Or just stop one job until human input
      

• Performs cleanups of crash data
  
  • Maybe automatically?
    
  • Maybe only when human says to proceed?
    

• Kinds of reports
  
  • Individual job run
    
    • Datetime of start, end
      
    • Status
      
    • Parameters used
      
    • Directory in which it started
      
    • Environment when it started
      
    • Account under which it ran
      
  • Job reports
    
    • By job
      
    • By date range
      
    • By category
      
  • Job control tables
    
    • Lists of jobs in each table
      
    • Lists of jobs/chains in each chain
      
    • Links to detailed info about each one
      
  • User reports
    
    • Distribution lists
      
    • Number of emails sent?
      

• Information available for reporting
  
  • List of job runs
    
  • Start and end times, duration
    
  • Success
    
  • Statistics
    
    • Example
      
      • Publishing ran 26 times in May
        
      • Average run time = 02:16:21 (H:M:S)
        
      • Max run time = 15:52:11 at 05:15:2016
        
      • Min run time = 00:00:41 at 05:13:2016
        
      • Successes = 24
        
      • Failures = 1
        
      • Jobs killed = 1