CDR Tickets

BZDATETIME::2011-10-26 12:10:50
BZCOMMENTOR::Bob Kline
BZCOMMENT::1

Alan, Volker, and I have been meeting to discuss the issues involved with this task. There are still some technical hurdles to be addressed, so it's unlikely we'll have anything to present by tomorrow's status meeting.

BZDATETIME::2011-11-02 16:47:46
BZCOMMENTOR::Bob Kline
BZCOMMENT::2

We have had several meetings to go over the technical issues for this task. One of the biggest hurdles is handling the edge case in which a module with tables or figures is included more than once. If we allow this, and the user must be able to control which instance of an included table or figure is the target of a reference, then the problems are at least an order of magnitude more difficult. Somewhat less challenging would be allowing the inclusion of the same module multiple times, but having the references to the included tables or figures refer to the first instance of those targets. The most straightforward path to implementing dynamic table and figure numbering would be to disallow multiple inclusions of the same module. If the users have a very compelling use case for multiple inclusions of the same module, it would be good to present it at the next status meeting.

In the meantime, the general approach for the straightforward implementation would involve a new pair of elements: <TableNumber/> and <FigureNumber/>. These would be used both in the tables or figures themselves, as well as in the references to those tables and figures. An example:

Summary module CDR0000000001:
<Summary>
:
<Table cdr:id="#_1><Title>Table <TableNumber/>. Toenail Shapes</Title>
:
</Table>
:
</Summary>

Base summary document CDR0000000002:

<Summary>
:
<Para...>... See
<SummaryFragmentRef cdr:href="CDR0000000001"#_1">Table
<TableNumber/></SummaryFragmentRef> for more information...
</Para>
<SummaryModuleLink cdr:ref="CDR0000000001">...</SummaryModuleLink>
:
<Table cdr:id="#_1><Title>Table <TableNumber/>. Mortality Rates</Title>
:
</Table>
<Para...>...
<SummaryFragmentRef cdr:href="CDR0000000002"#_1">Table
<TableNumber/></SummaryFragmentRef> contains interesting statistics
...
</Para>
:
</Summary>

The document as published:

<Summary>
:
<Para...>... See <SummaryRef href="CDR0000000001#_1">Table 1</SummaryRef>
for more information...</Para>
:
<Table id="CDR0000000001#_1"><Title>Table 1. Toenail Shapes</Title>
:
</Table>
:
<Table id="CDR0000000002#_1"><Title>Table 2. Mortality Rates</Title>
:
</Table>
<Para...>...
<SummaryRef href="CDR0000000002#_1">Table 2</SummaryRef> contains
interesting statistics...
</Para>
:
</Summary>

BZDATETIME::2011-11-02 18:15:09
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

(In reply to comment #2)

> ...
> In the meantime, the general approach for the straightforward implementation
> would involve a new pair of elements: <TableNumber/> and <FigureNumber/>.
> ...

Thinking more about this, there is another option which I just discussed with Bob and thought worthy of recording the discussion.

In comment #2 the proposal is to have a TableNumber element inside a SummaryFragmentRef and another inside a Table, and similarly for Figures.

An alternative is to create four new elements. Name might be something like:

TableNumber (inside a Table)
TableSeen (inside a SummaryFragmentRef)

and similarly for FigureNumber and FigureSeen. (Perhaps "TableSeen" is not the best name. I thought "TableNumberRef" might be good, but Bob pointed out that that is a different and possibly confusing use of the "Ref" suffix from our usual use.)

This might simplify the software to some extent. One filter pass could populate all table numbers, for example:

<TableNumber/>
...
<TableNumber/>

becomes:

<TableNumber>1</TableNumber>
...
<TableNumber>2</TableNumber>

A second pass then resolves all of the TableSeen elements so that:

<TableSeen/>

becomes:

1 (or whatever value corresponds to the associated cdr:href)

The choice between this and Bob's original approach might take into consideration at least the following:

1. Is it easier or harder for a user to understand?

2. Is it easier or harder for Volker to implement in the publishing filters?

BZDATETIME::2011-11-03 10:03:22
BZCOMMENTOR::Volker Englisch
BZCOMMENT::4

(In reply to comment #3)
> 2. Is it easier or harder for Volker to implement in the publishing filters?

I like the idea of using two different elements but I think the programming effort for both options is probably very similar with slightly less work for the two element option.
How about TableNumberLbl instead of TableSeen (I would prefer TableNumberRef, too, but I see the problem the "Ref" would create).

BZDATETIME::2011-11-03 15:39:58
BZCOMMENTOR::Bob Kline
BZCOMMENT::5

One other feature I neglected to mention or include in my examples was the ability to exclude a table or figure from the numbering sequence. We will implement this feature with an optional attribute for the Table element and the MediaLink elements. We can name the attribute "ExcludeFromNumberingSequence" and a valid value of "Y" (or something like that).

BZDATETIME::2011-11-16 11:39:06
BZCOMMENTOR::Bob Kline
BZCOMMENT::6

(In reply to comment #2)

> One of the biggest hurdles is handling the edge case in which a module with
> tables or figures is included more than once.

Did we decide in last Thursday's meeting that this would not be allowed, or did you want to check with the other board managers?

BZDATETIME::2011-11-16 15:33:01
BZCOMMENTOR::Margaret Beckwith
BZCOMMENT::7

Yes, we decided that we would not use modules more than once in the same summary.

BZDATETIME::2011-11-17 09:33:19
BZCOMMENTOR::Bob Kline
BZCOMMENT::8

I propose that we adopt Alan's suggestion of having four new elements instead of two, implement the schema changes on Mahler, mock up some summary documents, and have Volker do some proof-of-concept implementation of the XSL/T. Let's use:

• TableNumber

• ReferencedTableNumber

• FigureNumber

• ReferencedFigureNumber

BZDATETIME::2011-11-22 08:46:21
BZCOMMENTOR::Bob Kline
BZCOMMENT::9

(In reply to comment #8)
> I propose that we adopt Alan's suggestion of having four new elements instead
> of two, implement the schema changes on Mahler, mock up some summary documents,
> and have Volker do some proof-of-concept implementation of the XSL/T. Let's
> use:
>
> * TableNumber
> * ReferencedTableNumber
> * FigureNumber
> * ReferencedFigureNumber

[X] Change schemas on Mahler
[ ] Mock up test summary documents
[ ] Implement/test proof-of-concept filter changes

BZDATETIME::2011-12-08 13:52:52
BZCOMMENTOR::Bob Kline
BZCOMMENT::10

Ready for Volker's part.

BZDATETIME::2011-12-09 16:25:48
BZCOMMENTOR::Volker Englisch
BZCOMMENT::11

(In reply to comment #9)
> [X] Change schemas on Mahler

Bob, I don't see these new elements in XMetaL.
Does the XMetaL DTD still need to be updated?

BZDATETIME::2011-12-09 22:34:12
BZCOMMENTOR::Bob Kline
BZCOMMENT::12

(In reply to comment #11)

> Bob, I don't see these new elements in XMetaL.
> Does the XMetaL DTD still need to be updated?

I see the new elements in the schema and in the DTD on Mahler. I did notice, though, when I was investigating in response to your question, that Margaret saved the schema at the end of the day yesterday. What changes did you make, Margaret?

BZDATETIME::2011-12-12 11:04:59
BZCOMMENTOR::Volker Englisch
BZCOMMENT::13

I do see the FigureNumber/TableNumber available within the Para element but I thought these elements would be used within a Figure/Table and we'd be using the ReferencedFigureNumber/ReferencedTableNumber within Paras.

BZDATETIME::2011-12-12 12:04:59
BZCOMMENTOR::Bob Kline
BZCOMMENT::14

(In reply to comment #13)
> I do see the FigureNumber/TableNumber available within the Para element but I
> thought these elements would be used within a Figure/Table and we'd be using
> the ReferencedFigureNumber/ReferencedTableNumber within Paras.

Well, I think it works as is for figures. There you would put the placeholder in the Caption element, which allows the FigureNumber child to be used. If at some point the users asked you to prefix the caption with a figure number in the published output (perhaps you do that already?) you wouldn't need a placeholder at all.

For tables, I see that what I have done still doesn't allow the use of TableNumber inside the Title child of Table. I'll move TableNumber from OtherPhraseElements to TitleMarkupElements unless the users need me to create a separate complex type for TableTitleMarkupElements. What's your preference, Robin?

BZDATETIME::2011-12-12 13:44:40
BZCOMMENTOR::Robin Juthe
BZCOMMENT::15

(In reply to comment #14)
> (In reply to comment #13)
> > I do see the FigureNumber/TableNumber available within the Para element but I
> > thought these elements would be used within a Figure/Table and we'd be using
> > the ReferencedFigureNumber/ReferencedTableNumber within Paras.
> Well, I think it works as is for figures. There you would put the placeholder
> in the Caption element, which allows the FigureNumber child to be used. If at
> some point the users asked you to prefix the caption with a figure number in
> the published output (perhaps you do that already?) you wouldn't need a
> placeholder at all.
> For tables, I see that what I have done still doesn't allow the use of
> TableNumber inside the Title child of Table. I'll move TableNumber from
> OtherPhraseElements to TitleMarkupElements unless the users need me to create a
> separate complex type for TableTitleMarkupElements. What's your preference,
> Robin?

Is it labor-intensive to create a separate complex for TableTitleMarkupElements? It seems that would be cleanest and limit confusion and room for error.

BZDATETIME::2011-12-12 14:47:57
BZCOMMENTOR::Bob Kline
BZCOMMENT::16

I can make a new complex type, though it would apply to all Title elements (since we're using the same element name for Title elements which are children of Table as for those which are children of Section or SummarySection (and some others). The only way (other than replacing our home-grown schema validator with one of the third-party validation package which came into being after the CDR project was under way) would be to rename the Title child of Table to something like TableTitle, which would involve a bunch of rewriting of filters and reports, as well as some global changes.

One side effect of creating a separate complex type for titles is that you will restrict the ability to add the TableNumber element to only direct children of the table's Title element (so ...

<Table>
<Title><Strong>Table <TableNumber/></Strong>: ...

... wouldn't be allowed).

Do you still want a separate complex type?

BZDATETIME::2011-12-12 14:56:33
BZCOMMENTOR::Robin Juthe
BZCOMMENT::17

(In reply to comment #16)
> I can make a new complex type, though it would apply to all Title elements
> (since we're using the same element name for Title elements which are children
> of Table as for those which are children of Section or SummarySection (and some
> others). The only way (other than replacing our home-grown schema validator
> with one of the third-party validation package which came into being after the
> CDR project was under way) would be to rename the Title child of Table to
> something like TableTitle, which would involve a bunch of rewriting of filters
> and reports, as well as some global changes.
> One side effect of creating a separate complex type for titles is that you will
> restrict the ability to add the TableNumber element to only direct children of
> the table's Title element (so ...
> <Table>
> <Title><Strong>Table <TableNumber/></Strong>: ...
> ... wouldn't be allowed).
> Do you still want a separate complex type?

Thanks for the explanation. It sounds like it makes more sense to put it in the TitleMarkupElements complex. So, let's do that.

BZDATETIME::2011-12-12 15:22:00
BZCOMMENTOR::Bob Kline
BZCOMMENT::18

OK, TableNumber has been moved on Mahler. Ready for testing.

BZDATETIME::2011-12-14 16:37:57
BZCOMMENTOR::Volker Englisch
BZCOMMENT::19

Bob, I don't think the ReferencedTableNumber and ReferencedFigureNumber elements are setup properly. Shouldn't these elements be children of the SummaryFragmentRef elements? Otherwise I don't see how we can link the ReferencedTableNumber element with the corresponding table with the TableNumber element.

BZDATETIME::2011-12-14 18:27:15
BZCOMMENTOR::Volker Englisch
BZCOMMENT::20

Why are we using the Table attribute ExcludeFromNumberingSequence?

First of all, wouldn't it be simpler to add this attribute to the TableNumber element itself and secondly, would it be enough to not use the TableNumber element within the Title in order to exclude it from numbering?

The template that I have written counts the number of previously encountered TableNumber elements plus 1 in order to display the table number.

(By the way, in my opinion the attribute name is too long: ExcludeNumbering would probably be sufficient.)

BZDATETIME::2011-12-14 20:15:46
BZCOMMENTOR::Bob Kline
BZCOMMENT::21

(In reply to comment #19)
> Bob, I don't think the ReferencedTableNumber and ReferencedFigureNumber
> elements are setup properly. Shouldn't these elements be children of the
> SummaryFragmentRef elements? Otherwise I don't see how we can link the
> ReferencedTableNumber element with the corresponding table with the TableNumber
> element.

You're right. I didn't realize the SummaryFragmentRef didn't already allow other markup. I've moved those two elements.

(In reply to comment #20)
> Why are we using the Table attribute ExcludeFromNumberingSequence?
>
> First of all, wouldn't it be simpler to add this attribute to the TableNumber
> element itself and secondly, would it be enough to not use the TableNumber
> element within the Title in order to exclude it from numbering?

It's conceivable a user would prefer not to include the table number in the string appearing next to the table, but still wants it counted. We can discuss the question in tomorrow's meeting.

> (By the way, in my opinion the attribute name is too long: ExcludeNumbering
> would probably be sufficient.)

Sure. Why not "Unnumbered"? That's even shorter (and possibly a teensy bit closer to the mark).

BZDATETIME::2011-12-15 09:59:26
BZCOMMENTOR::Volker Englisch
BZCOMMENT::22

(In reply to comment #21)
> You're right. I didn't realize the SummaryFragmentRef didn't already allow
> other markup. I've moved those two elements.

I had another thought about this:
You said it could be possible the user might want to count a table within a sequence but not display the number.
Similarly, it could be possible the user would want to refer to a table or figure but not necessarily link to it. In this case the user would want to list the ReferencedTableNumber outside of the SummaryFragmentRef.

BZDATETIME::2011-12-15 11:36:54
BZCOMMENTOR::Bob Kline
BZCOMMENT::23

(In reply to comment #22)

> Similarly, it could be possible the user would want to refer to a table or
> figure but not necessarily link to it. In this case the user would want to
> list the ReferencedTableNumber outside of the SummaryFragmentRef.

OK, I've added these back to the OtherPhraseElements group.

BZDATETIME::2011-12-19 11:11:37
BZCOMMENTOR::Robin Juthe
BZCOMMENT::24

I marked up a messy test document on Mahler.

CDR0000062884 - Merkel Cell Treatment summary

The document includes the following:
-several tables and figures
-a module (contained in CDR696926) which includes a table and a figure
-references to several tables and figures, including references to the table and figure contained inside the module
-an unnumbered table containing two unnumbered images

Is there anything else that would be helpful to test in this document?

The document also contains something I wasn't expecting - a table in an image. See "Merkel - Immunohistochemical differential diagnosis". Since this is taken directly from a journal, the title (and table #) from the journal is on the image. I don't think there's much we can do about this, short of re-creating the table ourselves, but it has so many columns that I believe that's why it hasn't been done. I'm thinking we will just want to skip numbering of this entirely (either as a table or a figure), but we should talk about it.

In last week's CDR meeting, we discussed having "Table_X." added automatically to the beginning of the title string when the TableNumber element is used. After marking up this document I suggest we reverse that decision. Having the word "Table" at the beginning of the title makes it much easier to use the Find function to locate a table (or tables) in a summary. I don't think we want to take away that capability. So, I kept the "Table_" before the TableNumber element and the "." after the element in the document I marked up. (Same goes for Figures.)

We also discussed the need to remove the filler text that automatically populates each of these elements when they're entered into the summary. Just wanted to capture that decision in the issue.

BZDATETIME::2011-12-19 16:08:33
BZCOMMENTOR::Volker Englisch
BZCOMMENT::25

I thought I had submitted this comment on Friday but it's still sitting here:

I am at a point where I can now number the tables in sequence - including tables included from modules - and link to the table.
Next I will need to work on linking/displaying the table number coming from a module and I need to be able to create a table number outside of a SummaryFragmentRef.

I would also suggest to add some CSS for the new elements so that users can see some label where table numbers are being substituted.

BZDATETIME::2011-12-19 16:37:34
BZCOMMENTOR::Volker Englisch
BZCOMMENT::26

I've updated the CSS to add a "[#]" as a label for the FigureNumber and TableNumber elements. The documents updated where
Summary.css
Summary_structure.css
Summary.ctm

BZDATETIME::2011-12-21 15:53:01
BZCOMMENTOR::Volker Englisch
BZCOMMENT::27

In order to test the dynamic Table and Figure numbering Robin created a test document which is great because it includes all sorts of links - within a summary, from module to summary, from summary to module, etc. - and it allows me to resolve all of the problems for a given scenario.

The numbering schema implemented earlier had worked fine with my simple test document but failed to correctly number the tables and figures with Robin's complex test document, so I had to rework the numbering.

I also had to rework the new filter denormalizing module information since I did not correctly account for module summaries included on a sub-section level.

Currently, there is one more scenario of links that I will have to address. In addition, we will need to adjust the QC reports because a SummaryFragmentRef to a document that is not the current document currently causes a new QC report for that summary to be created. However, when the link goes to a location within an included module we don't want to treat that module as a different summary and rather want to jump to the location within the document instead of calling the QC report for the module.

BZDATETIME::2011-12-22 17:25:03
BZCOMMENTOR::Volker Englisch
BZCOMMENT::28

A SummaryFragmentRef is created as
<SummaryFragmentRef cdr:ref="CDR0000012345#_111"/>

When we are including a Summary module we have five possible scenarios of what this SummaryFragmentRef may be linking to:
1) The element might exist in the main document and link to a location in
the document itself.
2) The element might exist in the main document and link to a location in
the module.
3) The element might exist in the module and link to a location within the
module.
4) The element might exist in the module and link to a location within the
main document
5) The element might link to an external (non-module) document.

I had to adjust the cdr:ref and cdr:id attributes such that Cancer.gov will still "think" that these links are part of the document itself, which means the element
<SummaryFragmentRef cdr:href="CDR0000099999#_999"/>
will have to be modified to appear to be coming from the main document such as
<SummaryFragmentRef cdr:href="CDR0000012345#_<something>"/>
where <something> will need to be unique across the assembled document. I have done this and I'm now able to handle the cases (1)-(4) by adjusting the element to
<SummaryFragmentRef cdr:href="CDR0000012345#_sm_CDR000009999_999"/>

Next I will need to adjust case (5) to ensure proper linking to a summary document that is neither itself, nor a module included within the document.

Question: If a Summary Module is included within a document and is published
on it's own should a SummaryFragmentRef link to that external
document or link to the internal information?

BZDATETIME::2012-01-03 13:44:14
BZCOMMENTOR::Robin Juthe
BZCOMMENT::29

(In reply to comment #28)
> Question: If a Summary Module is included within a document and is published
> on it's own should a SummaryFragmentRef link to that external
> document or link to the internal information?

Margaret and I discussed this today and we're still a little confused as to whether this question (and the question about links to tables and figures inside a module) refer to links on QC reports or links on Cancer.gov (or both)? Thanks.

We also think it would be helpful to get some data regarding the number of tables, figures, and references to tables and figures to help us in planning how we will roll out this new functionality once it's ready (as was suggested in last week's CDR meeting). I'll put in a separate issue for that task.

BZDATETIME::2012-01-03 14:03:22
BZCOMMENTOR::Volker Englisch
BZCOMMENT::30

(In reply to comment #29)
> Margaret and I discussed this today and we're still a little confused as to
> whether this question (and the question about links to tables and figures
> inside a module) refer to links on QC reports or links on Cancer.gov
> (or both)?

Both.

The links and targets for both, the QC reports and Cancer.gov are build very similar and depend on the XML created, so it is a matter of how I'm creating these links with the filter.

BZDATETIME::2012-01-10 17:27:26
BZCOMMENTOR::Volker Englisch
BZCOMMENT::31

Bob, I believe we had said that we want to be able to use a table/figure number in the text without necessarily linking to it (meaning outside of a SummaryFragmentRef) and we wanted to provide an attribute that would be used to identify the table/figure id to create the sequence number.

We still will need to create this attribute, right?

BZDATETIME::2012-01-11 08:35:59
BZCOMMENTOR::Bob Kline
BZCOMMENT::32

(In reply to comment #31)
> Bob, I believe we had said that we want to be able to use a table/figure number
> in the text without necessarily linking to it (meaning outside of a
> SummaryFragmentRef) and we wanted to provide an attribute that would be used to
> identify the table/figure id to create the sequence number.
>
> We still will need to create this attribute, right?

I think the schema already supports this:

<Para>As we have seen in Table <ReferencedTableNumber
Target="CDR0000634279#_42"/> the outcomes for toenail cancer are ...</Para>

BZDATETIME::2012-01-11 08:41:30
BZCOMMENTOR::Bob Kline
BZCOMMENT::33

Looks like the schema had the support but the DTD for XMetaL wasn't updated. Should work (on Mahler) now.

BZDATETIME::2012-01-18 13:57:06
BZCOMMENTOR::Volker Englisch
BZCOMMENT::34

I'm guessing we would probably need a new macro or modify the existing one that allows the users to copy/paste these fragment links if we don't want them to manually edit the attributes of the ReferencedTableNumber/ReferencedFigureNumber.

The existing copy/paste FragmentRef macro currently doesn't work for this purpose.

BZDATETIME::2012-01-18 15:44:44
BZCOMMENTOR::Volker Englisch
BZCOMMENT::35

I've had so much fun with this I think it's time to share.

I've added several more links to our test documents (Merkel Cell Carcinome Treatment - CDR62884; and Gnome-Wide Searches Module - CDR696926) to cover the different scenarios we're going to encounter
a) ReferencedTableNumber/ReferencedFigureNumber within SummaryFragmentRef
i) Link from Main to Module
ii) Link from Main to Main
iii) Link from Module to Main
iv) Link from Module to Module
b) ReferencedTableNumber/ReferencedFigureNumber with Target attribute
i) Reference from Main to Module
ii) Reference from Main to Main
iii) Reference from Module to Main
iv) Reference from Module to Module

There are now 4 new filters:
CDR712005 - Denormalization Filter: Summary Module
Mahler: CDR696900
CDR723649 - Denormalization Filter: Summary Module IDs
Mahler: CDR696930
CDR723518 - Denormalization Filter: Summary Count Tables and Figures
Mahler: CDR696928
CDR723519 - Denormalization Filter: Summary Resolve Table and Figure Number
Mahler: CDR696929

I also created two new Filter sets for testing. This is necessary in order to run the regular summary filters while working with these new filters. The two new Filter Sets are
Denormalization Summary with Module Set
QC Summary with Module Set

This is ready for QC testing on MAHLER.

We will still need to test the publishable output separately but we'll have to wait until publish preview has been updated on GatekeeperQA.

BZDATETIME::2012-01-20 15:51:25
BZCOMMENTOR::Volker Englisch
BZCOMMENT::36

I've copied all of the modified filters to FRANCK and ran a diff report.
There were no differences between the two sets of summary documents. This means the filters can be copied to production and will still produce the same output as long as none of the new Table/Figure numbering elements have been used.

BZDATETIME::2012-01-20 15:59:09
BZCOMMENTOR::Bob Kline
BZCOMMENT::37

Hurrah! Well done, and thanks!

BZDATETIME::2012-01-23 15:57:49
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::38

Before I start to create my own test documents, I have been reviewing pup preview of the test document CDR0000062884 from Mahler. The table and figure references are displaying but I do not see actual numbers like Figure 1 or Table 3 or Table Distant Metastases etc. in the references and it looks like the tables and figures themselves are not numbered in pub preview. Am I looking at this the right way?

BZDATETIME::2012-01-23 16:11:13
BZCOMMENTOR::Volker Englisch
BZCOMMENT::39

(In reply to comment #38)
> Am I looking at this the right way?

Unfortunately you would need to look at it some other way.

PublishPreview is using the 'Vendor Summary Set' output. Because I need to be able to provide Summaries data to Gatekeeper I wasn't able to make changes to this filter set and I've created a new filter set called 'Vendor Summary with Module Set'.
On Mahler, the best you could do would be to run the Summary QC report manually by using the interface
CIAT -> Reports -> General Reports --> Filter Document
and enter for the Filter ID
'set:QC Summary with Module Set'

On FRANCK I have the new filters incorporated in the 'Vendor Summary Set' and you'll be able to run a PP report.

BZDATETIME::2012-01-24 14:25:09
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::40

(In reply to comment #39)
> (In reply to comment #38)
> On FRANCK I have the new filters incorporated in the 'Vendor Summary Set' and
> you'll be able to run a PP report.

It looks like the schema changes are not on Franck yet. Can you promote it to Franck for testing ?

BZDATETIME::2012-01-24 14:47:38
BZCOMMENTOR::Volker Englisch
BZCOMMENT::41

Bob, could you please update the schema on FRANCK to add the new elements to number figures and tables?

BZDATETIME::2012-01-24 16:47:45
BZCOMMENTOR::Bob Kline
BZCOMMENT::42

(In reply to comment #41)
> Bob, could you please update the schema on FRANCK to add the new elements to
> number figures and tables?

Done.

BZDATETIME::2012-01-25 14:56:48
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::43

On Franck, when I add the TableNumber elements and version the document, they get deleted after that.
Example :CDR0000445441

BZDATETIME::2012-01-25 15:12:22
BZCOMMENTOR::Volker Englisch
BZCOMMENT::44

I can't confirm that. I have no problem adding the new elements.
Something else must be going on. Have you tried to validate? Where are you adding the new elements?

BZDATETIME::2012-01-25 15:43:01
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::45

(In reply to comment #44)
> I can't confirm that. I have no problem adding the new elements.
> Something else must be going on. Have you tried to validate? Where are you
> adding the new elements?

1.Seems like something really odd is happening or I am doing something terribly wrong. I see the ones (elements) you just added (did you?) and I completed adding the elements to the rest of the tables in the sample document. I saved, versioned and validated the document. All the ones I added have been deleted. Did you add the elements to 2 tables (I think that is what I saw)? It seems like the one you added to the second table is gone. Only the first one exists.
I added the elements in the same place that you added to the other tables (first 2 tables), inside the Title element and just after the "Table" text. When I add the TableNumber element, there is usually "TableNumber" text in brackets. Am I supposed to do anything with it? Also, why is this different from what we had in Mahler where there was only the # sign?

2. I see that you saved and versioned your changes. I saved, versioned and validated my changes.

BZDATETIME::2012-01-25 15:51:07
BZCOMMENTOR::Volker Englisch
BZCOMMENT::46

(In reply to comment #45)
> I see the ones (elements) you just added (did you?)

Yes.

> When I add the TableNumber element, there is usually "TableNumber" text in
> brackets. Am I supposed to do anything with it?

Delete it. That's the only difference between the first element I had entered and the second. I believe the elements that are empty with the default {..} text will be removed.

> Also, why is this different from what we had in Mahler where there
> was only the # sign?

The CSS probably hasn't been copied yet. I'll check it out and change that.

BZDATETIME::2012-01-27 12:23:28
BZCOMMENTOR::Volker Englisch
BZCOMMENT::47

(In reply to comment #39)
> On FRANCK I have the new filters incorporated in the 'Vendor Summary Set' and
> you'll be able to run a PP report.

I had forgotten to copy one more filter change to FRANCK:
CDR335424 - Denormalization Filter: Summary - R10307

BZDATETIME::2012-01-27 14:41:35
BZCOMMENTOR::Robin Juthe
BZCOMMENT::48

The numbering of tables and figures both within a main summary document and inside a module that is contained in the document both seem to be working well, but this will need more testing. I know these changes are being promoted today as part of a bunch of filter changes, but these new elements are not ready to be used. We will continue testing and promote additional changes if needed.

One thing that is consistently happening in my testing is that links to tables and figures inside a module (either from the main document or from the module included inside the main document) are bringing me to the stand-alone module, rather than the module that is included in that document.

BZDATETIME::2012-01-27 14:55:33
BZCOMMENTOR::Volker Englisch
BZCOMMENT::49

(In reply to comment #48)
> One thing that is consistently happening in my testing is that links to tables
> and figures inside a module (either from the main document or from the module
> included inside the main document) are bringing me to the stand-alone module,
> rather than the module that is included in that document.

Is this happening for the QC report or the publish preview report or both?

BZDATETIME::2012-01-27 15:11:42
BZCOMMENTOR::Robin Juthe
BZCOMMENT::50

(In reply to comment #49)
> (In reply to comment #48)
> > One thing that is consistently happening in my testing is that links to tables
> > and figures inside a module (either from the main document or from the module
> > included inside the main document) are bringing me to the stand-alone module,
> > rather than the module that is included in that document.
> Is this happening for the QC report or the publish preview report or both?

This is happening in the QC report.

The links to a table or figure inside a module are not doing anything in Publish Preview.

BZDATETIME::2012-01-27 15:39:18
BZCOMMENTOR::Volker Englisch
BZCOMMENT::51

Another filter had been updated but not copied to FRANCK:
CDR409593 - Copy XML for Summary Report - R10310

The problem of calling the module rather than staying within the document is now fixed on FRANCK.

BZDATETIME::2012-01-27 15:57:32
BZCOMMENTOR::Volker Englisch
BZCOMMENT::52

(In reply to comment #50)
> The links to a table or figure inside a module are not doing anything in
> Publish Preview.

The problem here is that a link is typically identified by the CDR-ID of the document and its target ID (the cdr:id attribute value). I handled this situation for the QC reports because I had control over the input and output.
I will have to discuss with Blair and Prasad how exactly Percussion is creating the links and then adjust either our filters or the PublishPreview program accordingly so that we can ensure that the links will work in the end on Cancer.gov as well.

BZDATETIME::2012-01-27 16:56:11
BZCOMMENTOR::Volker Englisch
BZCOMMENT::53

The following CSS files have been copied to FRANCK and BACH:
Summary.css - R10312
Summary_structurre.css - R10312
Summary.ctm - R10311

BZDATETIME::2012-01-27 22:17:18
BZCOMMENTOR::Volker Englisch
BZCOMMENT::54

Updating multiple related bugs at once:

The following filters have been updated on BACH:

• CDR000157 - Vendor Filter: Summary - R10300

• CDR271370 - Module: Vendor Filter Templates - R10264

• CDR712005 - Denormalization Filter: Summary Module - R10299

• CDR723649 - Denormalization Filter: Summary Module IDs - R10299

• CDR723518 - Denormalization Filter: Summary Count Tables and Figures

• R10299

• CDR723519 - Denormalization Filter: Summary Resolve Table and Figure Number

• R10308

• CDR335424 - Denormalization Filter: Summary - R10307

• CDR409593 - Copy XML for Summary Report - R10310

• CDR609947 - Vendor Filter: Convert CG to Public Data - R10309

The DTD has been updated:

• pdq.dtd - R10122

• pdqCG.dtd - R10306

The CSS has been updated:

• Summary.css - R10312

• Summary_structure.css - R10312

• Summary.ctm - R10311

The filter set 'Denormalization Summary Set' has been updated to include the following members:

• Denormalization Filter: Summary Add Board Member - Spanish

• Denormalization Filter: Summary Module

• Denormalization Filter: Summary Module IDs

• Denormalization Filter: Summary

• Denormalization Filter: Summary Citation Formatting

• Denormalization Filter: Summary Reference List

• Denormalization Filter: Summary Reference De-Dup

• Denormalization Filter: Summary Reference Numbering

• Denormalization Filter: Summary InLine Numbering

• Denormalization Filter: Summary Citation Wrapper Add

• Denormalization Filter: Summary Citation Wrapper Sort

• Denormalization Filter: Summary Count Tables and Figures

• Denormalization Filter: Summary Resolve Table and Figure Number

BZDATETIME::2012-02-01 09:05:49
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::55

Pub preview appears not be working for summaries on Franck. I am getting python script errors for all Summaries that I checked.

A problem occurred in a Python script.

d:\cdr\Log\tmphwtu63.html contains the description of this error. d:\python\lib\cgitb.py:173: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 value = pydoc.html.repr(getattr(evalue, name))

BZDATETIME::2012-02-01 11:56:45
BZCOMMENTOR::Volker Englisch
BZCOMMENT::56

This might have been a temporary problem because I don't have any problems to run PP for this summary on FRANCK at the moment.
Could you please try again?

BZDATETIME::2012-02-02 11:25:29
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::57

(In reply to comment #56)
> This might have been a temporary problem because I don't have any problems to
> run PP for this summary on FRANCK at the moment.
> Could you please try again?

It must have been a temporary issue. It is okay now.
I get "Current element cannot accept links" error when I attempt to paste copied fragment links from FigureNumber and TableNumber elements to ReferencedTableNumber and ReferencedFigureNumber elements that are not within SummaryFragmentRef elements. I tried updating the Linking table in Mahler but that did not solve the problem. Could you please take a look? This is both on Franck and Mahler.

BZDATETIME::2012-02-02 11:28:56
BZCOMMENTOR::Volker Englisch
BZCOMMENT::58

Since the copy/paste of the fragment IDs is done with a macro this error isn't really related to this issue. You are still able to enter the fragment IDs manually, right?

Let's discuss this at the meeting.

BZDATETIME::2012-02-02 18:29:09
BZCOMMENTOR::Volker Englisch
BZCOMMENT::59

Please enter a new task for Bob to modify the macro/CPP code to paste fragment links to the Target attribute of the ReferencedFigureNumber/ReferencedTableNumber elements.

Also CC me to that task since this requires additional JavaScript changes that I've already done.

BZDATETIME::2012-02-09 10:00:59
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::60

(In reply to comment #59)
> Please enter a new task for Bob to modify the macro/CPP code to paste fragment
> links to the Target attribute of the
> ReferencedFigureNumber/ReferencedTableNumber elements.
>
> Also CC me to that task since this requires additional JavaScript changes that
> I've already done.

Done - OCECDR-3477

BZDATETIME::2012-03-01 09:41:45
BZCOMMENTOR::Robin Juthe
BZCOMMENT::61

Margaret and I reviewed the modules and dynamic table and figure numbering together and it's looking really good! We noticed a few issues:

1. Text/tables within insertion elements inside a module are displayed in Publish Preview when the module is included as a part of another document.

2. Proposed changes inside a module are displaying on QC reports of the main document regardless of whether the option to show proposed markup is selected.

3. Links to tables & figures that have targets inside a module are not working in PP. (this was mentioned above in comment 52)

The documents we used for testing are CDR62884 (main document) and CDR714430 (module) on Franck.

BZDATETIME::2012-03-01 11:36:32
BZCOMMENTOR::Volker Englisch
BZCOMMENT::62

(In reply to comment #61)
> 1. Text/tables within insertion elements inside a module are displayed in
> Publish Preview when the module is included as a part of another document.

This situation is easily explained:
When we are running a PublishPreview report we're processing the document through the vendor filters which should not include any insertion/deletion markup. Therefore, as the very first step of the filter process, we're sending the document through a filter that strips out all of the insertion/deletion markup.
The next step of our filter process is to denormalize all links and pull in all documents or document fractions that need to be included. When we do this we're now "dirtying" the previously cleaned document with any markup tags that may be part of the newly included document.

A fix to this problem may be to simply reapply the "markup cleanup" filter but I'm not certain that this won't create other problems, so it needs to be investigated further and we may need to create a second "markup Cleanup filter which needs to be applied to included documents only.

> 2. Proposed changes inside a module are displaying on QC reports of the main
> document regardless of whether the option to show proposed markup is selected.

Very similar to the first issue: The very first thing in the filter process is to create proper markup wrapping nodes based on the markup-display options checked by the user. Again, this is done before the summary module has been included and therefore the markup within a module won't be displayed properly.

BZDATETIME::2012-03-01 18:26:29
BZCOMMENTOR::Volker Englisch
BZCOMMENT::63

(In reply to comment #62)
> A fix to this problem may be to simply reapply the "markup cleanup" filter

I ran a test on MAHLER where I modified the order of the individual filters in the filter set from:

• Remove the Insertion/Deletion markup (Revision Markup Filter)

• Denormalize Board Member Info

• Denormalize Module

• Rename Module cdr:ids

• Many more summary filters...

to

• Denormalize Board Member Info

• Denormalize Module

• Rename Module cdr:ids

• Remove the Insertion/Deletion markup (Revision Markup Filter)

• Many more summary filters...

I will need to investigate if this change would negatively affect any other output created using the Summary Denormalization Filter Set.
At the moment I'm running publishing diff reports on FRANCK to ensure the change won't break any of the vendor output for summaries without modules.

BZDATETIME::2012-03-02 13:45:57
BZCOMMENTOR::Volker Englisch
BZCOMMENT::64

(In reply to comment #61)
> 1. Text/tables within insertion elements inside a module are displayed in
> Publish Preview when the module is included as a part of another document.
>
> 2. Proposed changes inside a module are displaying on QC reports of the main
> document regardless of whether the option to show proposed markup is selected.

For both of these issues I was able to correct the problem by changing the order in which the summary filters are being called.
I had to move the two filters that denormalize the SummaryModule element out of the default 'Denormalization: Summary Set' filter and created an additional filter set called
Denormalization: Summary Module Set

In all of the vendor and QC Summary filter sets this new set had to be added as the very first filter set. This will ensure that the markup-cleanup filter (for the vendor output) and the Insertion/Deletion filters (for the QC reports) will apply their rules to the module content as well as the main document.

I'm currently running diff reports on FRANCK testing these changes.

BZDATETIME::2012-03-02 16:15:17
BZCOMMENTOR::Volker Englisch
BZCOMMENT::65

(In reply to comment #61)
> 3. Links to tables & figures that have targets inside a module are not working
> in PP. (this was mentioned above in comment 52)

This has been fixed on MAHLER and FRANCK.

BZDATETIME::2012-03-08 12:14:50
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::66

(In reply to comment #65)
> (In reply to comment #61)
> > 3. Links to tables & figures that have targets inside a module are not working
> > in PP. (this was mentioned above in comment 52)
>
> This has been fixed on MAHLER and FRANCK.

This seems to be working. The tables are numbered correctly. However, the table in proposed Revision level in the Summary Module is also displayed in pub preview. I am wondering if this right.

BZDATETIME::2012-03-09 12:19:02
BZCOMMENTOR::Volker Englisch
BZCOMMENT::67

(In reply to comment #66)
> However, the table in proposed Revision level in the Summary Module is also
> displayed in pub preview. I am wondering if this right.

Do you remember which document you used and on which server you were testing?

BZDATETIME::2012-03-09 12:25:19
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::68

(In reply to comment #67)
> (In reply to comment #66)
> > However, the table in proposed Revision level in the Summary Module is also
> > displayed in pub preview. I am wondering if this right.
> Do you remember which document you used and on which server you were testing?

"CDR62884 (main document) and CDR714430
(module) on Franck."

BZDATETIME::2012-03-09 13:19:57
BZCOMMENTOR::Volker Englisch
BZCOMMENT::69

Could you please have another look at the tables?

There are three tables in the module.

• The first table is in proposed deletion tags. This table is displayed
  in PP as it should (proposed deletions are not applied and therefore the
  table needs to be displayed)

• The second table is in proposed insertion tags. This table is not displayed
  in PP as is should (since the insertion is proposed the table is ignored)

• The third table isn't displayed with any markup and it's displayed properly.

Which of these three tables do you see as being displayed (or dropped) incorrectly from the PP report?

BZDATETIME::2012-03-09 13:56:11
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::70

(In reply to comment #69)
> Which of these three tables do you see as being displayed (or dropped)
> incorrectly from the PP report?

I just looked again and they all look fine. I didn't look too closely the last time and misinterpreted the proposed insertion as proposed deletion. Sorry about that.

BZDATETIME::2012-03-14 16:56:53
BZCOMMENTOR::Volker Englisch
BZCOMMENT::71

Sharon was the first brave soul trying out the new table and figure numbering automation. On Monday she published the summary CDR62896 (Childhood Acute Myeloid Leukemia) and I haven't heard about any problems at this point.

BZDATETIME::2012-03-15 14:26:18
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::72

We agreed in today's meeting to programmatically add a period after TableNumber element whenever it is added to a summary. We also said that Volker can proceed to make the changes and put it in production before tomorrow's publishing. However, Kirsten, who has just finished adding the element to tables in 15 summaries will not be able to clean up the periods after the changes because she has a Board Meeting tomorrow and she is currently busy working on tasks for Sharon. So I will suggest that we proceed to make the changes but not promote it to Bach until next week.

Also, we did not specifically talk about making the same changes to the FigureNumber element but I think it would be good to apply the same changes.

BZDATETIME::2012-03-15 16:36:44
BZCOMMENTOR::Volker Englisch
BZCOMMENT::73

(In reply to comment #72)
> We agreed in today's meeting to programmatically add a period after TableNumber
> element whenever it is added to a summary.

I will add a period plus a space. If the user adds an additional space it will be ignored by the browser.

BZDATETIME::2012-03-16 16:05:06
BZCOMMENTOR::Volker Englisch
BZCOMMENT::74

The following filters have been copied to FRANCK and BACH:
CDR271370 - Module: Vendor Filter Templates - R10347
CDR000157 - Vendor Filter: Summary - R10347

I've also updated the filter sets on BACH to make the adjustment for the QC reports to display insertion/deletion markup for content within summary modules.
The following filter sets have been created or adjusted:

• Denormalization Summary Module Set (new)

• Denormalization Summary Set

• QC QD Summary Patient Set

• QC QD Summary Patient Set (Bold/Underline)

• QC QD Summary Set

• QC QD Summary Set (Bold/Underline)

• QC Summary Patient Set

• QC Summary Patient Set (Bold/Underline)

• QC Summary Set

• QC Summary Set (Bold/Underline)

• Vendor QC Summary Set

• Vendor Summary Set

Please verify the changes on Cancer.gov after tonight's publishing job.

BZDATETIME::2012-03-20 15:47:27
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::75

(In reply to comment #73)
> (In reply to comment #72)
> > We agreed in today's meeting to programmatically add a period after TableNumber
> > element whenever it is added to a summary.
>
> I will add a period plus a space. If the user adds an additional space it will
> be ignored by the browser.

Sounds good. We are ready so please proceed to make the changes.

BZDATETIME::2012-03-21 18:50:59
BZCOMMENTOR::Volker Englisch
BZCOMMENT::76

The following filters have been updated on MAHLER and FRANCK:
CDR723519 - Denormalization Filter: Summary Resolve Table and Figure Number
R10364

Please note: I was not able to test summaries with figures due to a Gatekeeper
error.

BZDATETIME::2012-03-22 10:54:30
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::77

(In reply to comment #76)
> The following filters have been updated on MAHLER and FRANCK:
> CDR723519 - Denormalization Filter: Summary Resolve Table and Figure Number
> R10364
>
> Please note: I was not able to test summaries with figures due to a Gatekeeper
> error.

We are getting python script errors on Both Mahler and Franck:

A problem occurred in a Python script.

d:\cdr\Log\tmpliums4.html contains the description of this error. d:\python\lib\cgitb.py:173: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 value = pydoc.html.repr(getattr(evalue, name))

BZDATETIME::2012-03-22 11:04:42
BZCOMMENTOR::Volker Englisch
BZCOMMENT::78

(In reply to comment #77)
> > Please note: I was not able to test summaries with figures due to a
> > Gatekeeper error.
>
> We are getting python script errors on Both Mahler and Franck:

That's exactly why I included the note to the comment! You're not able to test the filter changes if your summary includes a figure.
I'll let you know once this has been resolved.

BZDATETIME::2012-03-22 18:50:54
BZCOMMENTOR::Volker Englisch
BZCOMMENT::79

As far as I can tell the PP report is still not working on MAHLER or FRANCK. However, it's possible to test the filter changes by using the QC reports, too.
The filter changes are replacing the given text for a Table or Figure like this
Table <TableNumber> This is a table title
to
Table 1. This is a table title

This replacement is the same regardless if you're running the PP report or one of the QC reports.

Since it's possible that the fix for the PP report will take a little longer I suggest to move the filter change to production (once you confirm it is working for QC reports) and then do a quick PP report check on BACH rather than not being able to move the change to production before the weekly publishing job runs. After all, the changes were very minor - 2 lines, one for tables, one for figures.

BZDATETIME::2012-03-23 07:53:50
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::80

(In reply to comment #79)
> As far as I can tell the PP report is still not working on MAHLER or FRANCK.
> However, it's possible to test the filter changes by using the QC reports, too.

Yes. We were able to see that it worked in the QC reports.

> The filter changes are replacing the given text for a Table or Figure like this
> Table <TableNumber> This is a table title
> to
> Table 1. This is a table title
>
> This replacement is the same regardless if you're running the PP report or one
> of the QC reports.
>
> Since it's possible that the fix for the PP report will take a little longer I
> suggest to move the filter change to production (once you confirm it is working
> for QC reports) and then do a quick PP report check on BACH rather than not
> being able to move the change to production before the weekly publishing job
> runs. After all, the changes were very minor - 2 lines, one for tables, one
> for figures.
This is fine with me but are you saying that PP will work in Bach when you move the changes into production ?

BZDATETIME::2012-03-23 10:06:25
BZCOMMENTOR::Volker Englisch
BZCOMMENT::81

(In reply to comment #80)
> This is fine with me but are you saying that PP will work in Bach when you move
> the changes into production ?

The problem with PP on MAHLER and FRANCK is related to changes in Gatekeeper and Percussion. If PP works on BACH before the filter is updated it will also work after the filter has been updated.

BZDATETIME::2012-03-23 10:09:47
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::82

(In reply to comment #81)
> (In reply to comment #80)
> > This is fine with me but are you saying that PP will work in Bach when you move
> > the changes into production ?
>
> The problem with PP on MAHLER and FRANCK is related to changes in Gatekeeper
> and Percussion. If PP works on BACH before the filter is updated it will also
> work after the filter has been updated.

OK. In that case, please promote the changes to Bach.

BZDATETIME::2012-03-23 12:51:14
BZCOMMENTOR::Volker Englisch
BZCOMMENT::83

The following filter has been updated on BACH:
CDR723519 - Denormalization Filter: Summary Resolve Table and Figure Number
R10364

Please verify on BACH by running the PublishPreview report.

We will be monitoring this change after the weekly publishing job finished.

BZDATETIME::2012-03-26 12:42:02
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::84

We've checked some of the affected summaries on Cancer.gov and they all looked good.

BZDATETIME::2012-03-30 17:35:07
BZCOMMENTOR::Robin Juthe
BZCOMMENT::85

This is working well so far and has been named Sharon's "favorite CDR enhancement ever", for the record. :-)

Closing issue.