CDR Tickets

Our desired specs for this new report are attached. We'd like to have two "flavors" - one basic option for day-to-day maintenance and one more advanced option for historical/management purposes.

Decided in the status meeting to count revisions instead of summaries (see comment in Word document).

I'm working on this.

The first thing I'll do is write up a document summarizing my understanding of what the report has to do - what the user inputs, what data we extract and from where in the documents, what appears in the output, etc. Then someone can fix whatever misapprehensions I have before I write the code.

The two reports are different enough that I plan to write them as two separate reports, with two separate entries in the report menu for Management Reports, e.g.,

"Summaries Type of Change (Most Recent Change)"
"Summaries Type of Change (All Changes)"

I plan to write up my understanding of the first one first, and perhaps implement it, before starting the second one.

 Summaries Type of Change (Most Recent Change) - Report Specification
 --------------------------------------------------------------------

Here is my understanding of what I am to do for this report.  We need to
correct any errors and answer any open questions before I write the
code.


                                 Inputs
                                 ------

The report will have an input screen for users to enter the following
data to specify the report:
    
    PDQ Editorial Board
        A drop down select list.  Only a single selection allowed.

    Summary
        A drop down select list of Summary titles for the selected
        board.  Multiple selections allowed.

        "All" should be at the top of the list to allow a user to see
        the most recent changes for all Summaries in the board. 

        [Note: I plan find the titles with the cruder but much simpler
         technique of going back to the server after a board has been
         selected to construct a revised form that has all the right
         Summary titles in it rather than using the elegant but more
         complex AJAX technology used in Drupal and EBMS.]

    Type of change
        A list of checkboxes for the types of change allowed by the
        schema for the element:

            /Summary/TypeOfSummaryChange/TypeOfSummaryChangeValue

        Multiple selections are allowed.

        For each type of change checkbox, also include a checkbox for
        whether comments should be included or not.  If a type of change
        value is not checked, the comment checkbox associated with it
        will be ignored.

Input questions:

    Should we also ask for Audience or Language?


                                Outputs
                                -------

The output should be a report in HTML format with header information and
the following columns.  All data for the columns will come from the
current working documents rather than the last publishable versions.

    Summary
        Heading = "Summary".
        Content = Document title? - See "Output questions".

        The rows of the report will be sorted alphabetically by Summary
        document title.

    For each selected type of change:
        Heading = type of change, e.g., "Comprehensive revision".
        Content = Date associated with the change, if and only if that
                  type of change is the last change recorded in the
                  document.

        If Comments were requested, then a comment column follows the
        type of change colum:
            Heading = "Comment".
            Content = Concatenation of all comments associated with the
                      change, with a separator, e.g.:
                         "Here is the first comment / Second comment".
                      If there are no comments, the comment cell is
                      blank.

        Only the latest dated type of change will be reported.  If there
        is more than one with the same date, the last one encountered in
        the document will be reported.

Output questions:

    Can we use the full Summary document title as the value of the
    Summary column?  The Word specification shows abbreviated titles.  I
    don't know where we'd get those.

    Should the CDR ID be included?

    Should it be hyperlinked to the Summary itself?  If so, to what:
        Display of XML?
        QC Report (slow to create.)

    If a Summary has no TypeOfSummaryChange element, should we put in a
    row for it anyway, or just leave it out of the report?

Thanks for writing this up, Alan. This sounds pretty good. Here are our thoughts:

INPUTS

We would like to use a more standard (yet also more complicated) interface that we've used for other reports when selecting the summary (or summaries). Specifically, we'd like to use the "Single summary" and "All summaries" blocks that we used for the "Summaries TOC Lists" report. This would give us the flexibility to select summaries by language and audience, in response to your question.

OUTPUTS

If possible, we would like to show all changes on the most recent date, rather than the last one encountered in the case of more than one change logged on the same date. We often complete both a reformat and a comprehensive review on the same date, so one of these would be "lost" from the report output if only one of the changes for that date were to display.

To answer your questions:

Please use the full summary title (abbreviations were just my short-hand for the example).

Please add the CDR in parentheses after the summary title.

Please do not link the CDR ID to anything (XML wouldn't do most of us any good, and the QC report would take too long to come up - it would be faster to pull up the summary in the CDR).

Please enter an empty row for a summary without any type of change elements.

Thank you again! Let us know if you have any follow-up questions.

Got it.

Inputs:

I will use the same interface as in the first and third input fieldsets in the "Summaries ToC Lists", leaving out the middle fieldset that prompts for "Shared Options".

Outputs:

I'll incorporate your answers to my questions.

Thanks for the feedback.

I'm working on the report.

I've written a new utility function cdr.getSchemaEnumVals() to use in the report in cdr.py. It retrieves all of the enumerated values for any simpleType in any schema, e.g., for creating a picklist.

It's tested and checked in to subversion.

I had meant to complete the first report today but several meetings plus work on five other JIRA issues intervened and I didn't get to do any work on it. Maybe Tuesday.

If I search for all Summaries that link to the the Adult Treatment board, I see 180 documents.

If I exclude those that aren't active, the number drops to 155.

If I only take those that have a published version, it drops to 144.

I'm guessing that it's only the 144 that should show in this report. Is that right? Or should I use one of the more inclusive searches?

Robin,

I think I know the answer to this question but I'll ask anyway to be absolutely sure:

If a user has asked for only one type of change, e.g., "Comprehensive revision", we will NOT report that change unless it is the latest one. In other words, we're looking for the latest changes to the document, not the latest changes of the types requested in the report.

Right?

Also, if you answered the previous question (180 vs 155 vs 144) verbally, I don't have any notes about it. Could you confirm that the answer is 144 (or whatever it is)

Thanks.

Hi Alan,

Yes, please go with 144 (only include published summaries).

As for your second question, it depends which flavor of the report is run. In the first (basic) version, we are only interested in seeing the most recent change. In the second (management) version, we would like to see every change of a certain type for a given time frame.

Thanks!

Here are some questions about the output reports:

 1. Ordering documents on both basic and advanced reports.

    I haven't been able to tell from looking at the samples how to order
    the documents in the output.

    I would think it would be most useful to order by document title.
    That way if the user wants to see a particular Summary's changes,
    it's easy to find.

    Is that reasonable?
    
 2. Choosing types of change.

    We're going to have a part of the input parameter form that asks
    which types of change (there are currently four of them) should be
    selected and, for each one, whether comments are desired.  This is
    needed for the "Basic" report.

    The specs for the advanced report say to get all types of change and
    all comments.  I think it's probably a freebie if you want me to
    look at the checkboxes for types of change and comments and do the
    following:

        If there are no check marks, act as if all boxes are checked,
        i.e., get all types of changes and all comments.

        If there are any types of change checked, then do whatever that
        section of the form says to do.

        That makes it possible to produce an advanced report that just
        shows one, two, or three types of change instead of all of them.

    I think the software I've already written for the basic report can
    be applied here with little extra work, if you want that capability.
 
 3. Ordering of the changes within a document.

    I haven't been able to tell from looking at the sample "Order by
    Summary" advanced report what order the results should appear in.

    Possible options could be:

     a. Order by type of change, and within type of change by date,
        maybe with the latest on top.

     b. Order by date, latest first, letting types of change come out
        wherever they appear in date order.

    I'm imagining that the second (b) is the most useful one.  Is that
    right?

    I can, if needed, allow any ordering the user likes by putting up an
    input form element to get the information, but I'd rather not do
    that unless it's clearly useful.

Thanks.

I think this report is working.  I've tested a fair number of bad
inputs, good inputs, and even some good but strange inputs and it seems
to come out okay in all flavors of the report.

The formatting is excellent in some respects:

    Nicely formatted tables are produced.

    Alternate lines have different background shading.

    The user can choose Excel or HTML just by clicking a radio button.

These goodies are due to Bob's new Report facility that adds those
things to any report that calls his code.

However there are a few things that I'd like to do differently - mainly
having to do with text and spacing outside the tables.  I couldn't
figure out how to do those with Bob's module.  I'll show it to him so
that he can say:

    "That's easy, just do this." or

    "Oh, let me add that to the module." or maybe even

    "Why in the world would anyone want to do that?"

However, none of those things will affect report content.  So I'm
declaring this ready for testing.

What are some of the capabilities you're looking for?

My interests for this report are to put markup above the first table and between the tables in the report.

Above the table I would put the information that is now in the Banner and Sub-banner of the HTML report.

The captions for the tables themselves are reasonably explanatory and I might not need anything more above each one, but it would look nice to add some whitespace between them.

What would you think about adding header and footer options to both the Report class and the Report.Table class? The caller would pass an arbitrary string of html. Another interesting possibility might be to add margin options, though that is far more specialized and therefore less generally useful.

I don't know whether these should have any effect in the Excel format output or just be ignored. They could reasonably be ignored, or maybe we could find something useful to do with them. A conceivable possibility is to have something like "html_header", "html_footer", "sheet_header", "sheet_footer". But these are just ideas. I think it's better to keep things clean and not go overboard. If we think it's useful to have these for html only, it would make sense to use the option name "html_header", not implement a "sheet_header", but leave the namespace open for it if we ever change our minds.

The Table constructor now supports two new optional keyword arguments:

• html_callback_pre

• html_callback_post

The arguments take as their values a callback function or method which will be invoked before the table is added to the report, and after the table is added to the report, respectively. The function or method is passed two arguments, one for the Table object to which the callback is attached, and the other for the Page object which is being assembled for the report. The callback can refer to the table object if necessary to determine which table is about to be (or has just been) marked up, and can use the Page object's methods to insert additional HTML markup for the report. For example:

def my_callback(table, page):
    instructions = "Lengthy explanation of how the following table is to be interpreted ...."
    page.add(page.B.P(instructions, page.B.CLASS("instructions")))
    page.add_css("p.instructions { border: solid dashed #eee; margin: 5px auto; }")

Will this provide the required functionality?

That should do what I need. I'll install something using it tomorrow.

I just saw that there are some outstanding questions above (Mar 12 comment). Sorry - here are my answers:

1. Your assumption to order by title is good.

2. Let's keep this as is and retrieve all types of change and all comments for the advanced report. We can always modify this later on.

3. Your assumption for the sequence is correct - the sequence you propose in (b) (date, with most recent ToC on top) is most useful.

Thanks. Is this report ready for review on DEV?

Hi Alan,

When this report is ready, could you please add it to the following menu page?

OCCM Board Managers>Management Reports

Please call it the "Summaries Type of Change Report".

Thanks.

It is ready for review on Dev, though I plan to do a few cosmetic changes tonight.

I have installed the report in the menu system at:

  Admin System (choose any user type):
    CIAT/OCCM Staff (or whatever)
      Reports
        Summaries and Miscellaneous Documents
          Management QC Reports
            Summaries Type of Change

You left out some intervening choices in your menu description but otherwise it appears that I either read your mind or remotely influenced it (:^) and put the report in the place where you wanted it, the only difference being I didn't put the word "Report" on the end of the report name.

I'll do that if you like, though you might like it as is since all the choices on that page are reports.

It's okay to add it there, but I was actually hoping to get it up on the Board managers page.

Admin System
OCCM Board Managers
Management Reports
Summaries Type of Change

(It's fine to leave off "report" - we haven't been entirely consistent about that, but I agree with you that everything in the list is a report, so it's redundant.)

This is looking good!

I have a few comments/questions so far:

1. When I run the basic report, the comments seem to duplicate across the row. Only the comment relevant to the preceding type of change column should display. In other words, if I enter a ToC element for a "major change" with a comment, then that comment should only appear next to the major change column. Is this feasible?

2. For the advanced report, by type of change option, could you please add the total of each type of change in the header for each sub-report?

3. For the types of change block, would it be possible to add an option for "all types" and "all comments" and have those selected by default?

4. Is it possible to use the calendar buttons that we've added to other reports? (not essential)

5. I think the report types - Basic and Advanced - need some qualification. Could you please add parenthetical notes after each one? Here's what I propose: "Basic (Most Recent Change)" and "Advanced (All Changes)"

6. I'm finding the interface a little confusing, but I know you're still making cosmetic updates so those will probably help!

I've added the report to the report to the board managers page and left it in the other too.

I'll work on the other comments this morning.

1. ... the comments seem to duplicate across the row ...

   That was a blindingly obvious bug.  Don't know how I missed it
   (hysterical blindness perhaps.)

   Fixed.

2. .. add the total of each type of change in the header for each sub-report?

   Done.

3. ... add an option for "all types" and "all comments" ...

   I added a single checkbox for "All types and all comments".  I
   thought that if there were two the user might not know what to expect
   if "All types" were unchecked but "All comments" checked.

   If the "All types and all comments" box is checked, I'll get them all
   and ignore any other check marks.

   Done.

4. Is it possible to use the calendar buttons ...

   I'm going to investigate this, but thought I'd go ahead and put the
   rest up for examination.

   Not yet done.

5. ... I propose: "Basic (Most Recent Change)" and "Advanced (All Changes)"

   Done.

6. I'm finding the interface a little confusing.

   I started out with the interface for another report that was
   mentioned as a model in the specs, then stuck on more fields with
   chewing gum.

   Maybe I'd have done better to have started fresh.

   I don't want to make any changes without consulting with you.  We
   could talk next Tuesday, or by email or phone.

I've added calendar widgets to the form.

I also discovered a bug in my selection of Spanish language Summaries when specific boards are checked. I didn't notice it before because there don't seem to be any Spanish language Summaries with TypeOfSummaryChange elements. So when I got no results for a particular board, I assumed that was why.

I'll fix that real soon, but it's not fixed yet.

I verified the changes on DEV - all looks good except for the bug you already reported with the Spanish.

As for the interface, I think it may help if the second-to-last box is titled "Basic Report Options" and the last box is titled "Advanced Report Options". I keep selecting options in the Types of Change box and then later realize that those only apply to the basic report, so I think that could be confusing for others, too.

thanks!

I corrected the Spanish board selection problem.

I found no change data in the Spanish Summaries to test, but the fix is to a more general Summary selection routine in our core cdr.py library, and I had a test program that can test that. I think it's working.

I'll make the user interface changes today.

I made some user interface changes - an expansion of the ideas in Robin's comment. Please have a look.

I tested the Spanish selection and it's working correctly.

The interface also looks good. I have a few picky requests, if possible:

1. Please change "HTML" to "Web page" and "Excel" to "Excel Workbook". This will be consistent with the media permissions report and the board roster report, each of which now have these options, too.

2. Please swap the sequence of the subheading "Document Title or CDR-ID" to "CDR-ID or Document Title" to match the sequence of the fields beneath it.

3. Please remove the word "Spanish" in front of each Board name in the Spanish summary selections.

4. Could any selection other than "all" in the "Basic Report - Select Types of Change" section cause the "all" option to become unchecked?

5. Please remove the statement, "Fill in the following additional fields..." since the new heading now says essentially the same thing.

Thanks!

Picky people produce practically perfect products.

I'll see what I can do.

I made all five of the changes requested above.

Change number 4 could be greatly extended - for checking this and unchecking that based on what the values of other checkboxes were. I went beyond the specific request in number 4, handling clearing specific boxes if "all" is checked, and handling the types of change as well as the board selections. However I called a halt when it seemed that I had the most important interactions covered and wasn't sure it paid to spend more time writing and testing javascript for each place where it could be of possible use.

More is possible if desired, but I'll move on to other things unless and until someone requests more.

Thanks, Alan. This looks good. I like the changes you've made.

Verified on DEV. The Excel report only works in Chrome or Firefox (I think this is as expected).

I think we need to make a couple more changes to the interface for this report.

The options to select specific summaries and to select the audience and type of summary are used for both reports, not just the basic report (as the interface says now). Could we change these boxes to remove "Basic Report -" from the beginning of the name of the box?

In the future (next release maybe), it would be nice to have the selection of the "basic" or "advanced" report then initiate javascript (or whatever it is) to show you the options for that report only 🙂

Margaret also noticed something funny with the way the data appeared on the basic report. I'm attaching a screenshot, but there's one row (Genetics of Prostate Cancer) that seems to extend far off the right hand side of the page...

One more thing.

This could be saved for a future release, but we don't need to see the summary type and audience in the summary column of the report. It can just include the summary title and CDR ID. Since we can limit the report output by the summary audience and type, it doesn't seem necessary to include that information on the report, too.

Thanks!

... there's one row (Genetics of Prostate Cancer) that seems to extend far off the right hand side of the page...

Can we get the selections made when this report was generated, so we can reproduce the problem?

Margaret noticed a few more things:

1. When she runs it in Excel for Basic, All English (default values), she gets a Python script error (see attached screenshot from Margaret); I didn't get this error earlier in the week. Excel seems to work okay for the Advanced report.
2. If she select Basic, and put in a title of a summary (Milk Thistle), she get an error that tells her that there are 2 (HP and Patient), even though HP is selected in the next box down (see screen shot from Margaret)
3. It is confusing to call these Basic and Advanced; we need to come up with better names
4. Margaret would like to have the ability to remove comments from the Advanced report.

For now, in addition to the few things mentioned in earlier comments, I think we should troubleshoot the error in #1, remove the summary title field to resolve #2 temporarily, and keep the names as is.

In the next release, we should rename the report versions, work on additional enhancements to the interface, and create more options to show specific types of change with or without comments for the Advanced report.

Margaret's screenshots.

Sure - it was the Basic Report, Web Page, All English, Health Professional, All Changes & Comments.

Thanks, Bob!

I'll be in on Monday, Tuesday and Thursday next week so I'll try to get everything fixed on Monday.

I think the bug is fixed that put change records after the end of the table and caused the Excel format to produce the Python error.

I've also made some of the user interface changes and am working on more.

One change I equivocated on is the deletion of the two parts of a doc title after the semicolons. I removed the part after the second semicolon (summary audience) from all of the reports since a user must select one or other audience and cannot select both.

However if multiple boards are included, I left the part in that says "Treatment", "Supportive Care", "Complementary and alternative medicine", etc. I thought that if there is more than one board included, that might be needed to show what document is actually being reported. However it's trivial and no risk for me to take that out for all occurrences of the report if you don't want it.

I'll make more changes before I go home tonight. One change I don't plan to make is to add the logic to repaint the form based on what a user wants. I'm nervous about making such a significant change so close to going to production.

I've changed the logic of the report as follows:

  For choosing a specific Summary:

    If a user enters a CDR-ID, I'll fetch whatever record matches the
    ID, ignoring anything else in the input form - audience or language.
    I'll fetch it even if it's blocked.

    However if a user enters a title string, I'll only search Active
    documents, and I'll only match the specified audience.  That way
    either the HP or the Patient Summary will come up, but not both.  I
    don't bother doing that with language since it would be unusual for
    the English and Spanish titles to match and I don't want to make the
    user check the language box for Spanish before entering a Spanish
    string.  However that's easy to add if desired.

    Whether the report is "basic" or "advanced", I now allow a user to
    enter a specific CDR-ID or title.

  For selecting boards and change types:

    With one caveat, I've made the checkboxes that select boards and
    change types both now work for both the "basic" and "advanced"
    reports.

    The caveat is that I haven't made the comment checkboxes do anything
    in the advanced report.  To do that will take a little more time
    than I have tonight.  For now, all advanced reports show comments.

I've also streamlined the interface.

  There are now only two main sections:
  
    One for specifying whatever a user wants for either type of report.

    One for specifying parameters that only apply to the "advanced"
    reports.

    There is no longer a pair of radio buttons at the top for Basic or
    Advanced.  Instead, a basic report is assumed unless the user
    scrolls to the bottom.

    The section at the bottom starts off with a checkbox:

       [ ] Advanced Report

    If that box is checked, then an "Advanced" report is generated and
    the additional input controls are read to detemine how to produce
    it.

I'm hoping it's now easier to understand and to use.

Please give it another test.

This interface is much better - thanks!

I'm still testing but I've noticed a couple of things that I wanted to mention.

1. There seems to be a bug in the option to display specific changes and comments for the basic report. When I select Web page, HP, All English, and Comprehensive Revision (no comments), I get a blank column heading for "comprehensive revision" with empty space below (no dates). I will attach a screenshot. This seems to happen anytime a type of change is selected without its accompanying comments for the basic report.

For the advanced report, selecting a type of change without the accompanying comments generates the change column with dates and the comments column, but it is not populated. (I know you said you hadn't done anything about comments in the advanced report yet, so this is probably expected.)

2. Please remove the summary type after the summary title name in the report output. In most cases this is redundant with the title (e.g', "Colon Cancer Treatment; treatment").

It looks like I introduced a dumb bug after making all of the changes. Sorry.

I believe it's fixed now.

I verified that the bug was fixed. Thank you!

The following is still true, though.
For the advanced report, selecting a type of change without the accompanying comments generates the change column with dates and the comments column, but it is not populated. Would it be possible to remove the comments column if it is not selected?

Now that we've lost the parenthetical notes about the two different reports ("most recent change" and "all changes"), we need to make it clear that the "most recent change" report will be generated by default and that the "advanced" report is really the "all changes" report. In fact, I think we should just replace the word "Advanced" with "All Changes". Would it be possible to add a sentence or two to the top of the page to say that there are two versions? For example:

There are two versions of this report. By default, the Most Recent Change report will be generated. Below, you will find additional options for the All Changes report.

Margaret is also reviewing the ToC report so she may have additional comments.

I made prompt and heading changes in the spirit of the requested changes. If they can be improved, let me know.

For the comment columns, I had hesitated to delete them in the advanced reports because there that is difficult and requires more re-writing of the report and potentially funkier aesthetics than may be needed.

In the Advanced / All Changes reports, if ANY selected type of change has comments checked, I'll show a comments column for all rows.

This is necessary for the organization by Summary report because there is just one table. If a comment can appear in any row, there has to be a comment column.

The type of change organization is the tricky case. That report contains up to four separate tables. If some types of change have comments checked and some don't, I could revise the report to have two columns for some of the tables and three for the others. But that requires a higher level of effort to implement because of the way the report was written (more would have to be rewritten than for the way I did it.) And I'm not sure it would look better anyway. So I handled that as I did for the "By Summary" organization. If ANY Comments boxes are checked, all of the rows have three columns. If none are checked, all have two columns.

If that doesn't work for you, let me know and I'll bite the bullet and fix it.

This is much better. Thank you!

Margaret and I just reviewed it and we have a few final wording requests:

1. Please change "Type of Report" (the heading of the first box) to "Type of Change Report".

2. Please change "All Changes - Select Dates and Organization" to "Type of Change History Report".

3. Please change "Show All Changes" to "Show Type of Change History".

Done, in subversion and on QA.

Looks good!

Sorry, one more:

In the blurb at the top of the page, please change "check the box marked "Show All Changes" " to "check the box marked "Show Type of Change History" " to reflect this latest change.

Thanks.

Done, in subversion and on QA

Looks good! Thanks!

Verified on QA.

This report is working well on prod, but I noticed that the appearance of the report headings is different on prod than on QA. See attached comparison.

I believe this problem was related to IE compatibility mode.

Verified on prod.

I've confirmed that the programs are identical. I also checked using Firefox on the two servers and did not see the differences. So it looks like the IE browser modes are indeed the variable factor.