CDR Tickets

BZDATETIME::2010-09-01 15:39:40
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::1

Here is the text from Mary:

Creation of an attribute of active/not active for the TermTypeName element would enable CIAT staff to designate a term that is not to be used for indexing new trials or reindexing existing ones. The desired effect would be that these antiquated terms would not show up when staff are searching for terms to link within trial indexing, but their presence in existing trial documents would not make those documents invalid (as would happen if the term was designated “obsolete”).

BZDATETIME::2010-09-08 11:28:59
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::2

I am re-assigning this to Alan so that we can try his plan on Mahler or Franck.

BZDATETIME::2010-09-16 20:59:53
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

(In reply to comment #2)
> I am re-assigning this to Alan so that we can try his plan on
> Mahler or Franck.

My plan was to use the picklist limiting code in the link
definition tables to prevent "Obsolete term" TermTypeNames from
appearing in the picklist. The idea was:

/Term/TermType/TermTypeName == "Index term"
AND
/Term/TermType/TermTypeName -= "Obsolete term"

Unfortunately, that didn't work. The idea is sound but, alas,
the code in the server was not up to snuff. It generates SQL to
check for the "Index term" type name, and then adds SQL to say,
in effect:

Don't include this on the picklist unless there is a
TermTypeName <> 'Obsolete term' for the link target doc.

That doesn't work because there IS a TermTypeName not equal to
"Obsolete term", namely the one for "Index term".

What I need is this:

Don't include this on the picklist unless there is NOT a
TermTypeName = 'Obsolete term' for the link target doc.

It looks like a straightforward fix, but I have to study the code
carefully before applying it. It's fairly complex code
generating nested SQL statements to handle all possible
combinations of ANDs, ORs, NOTs, and parentheses.

I'll work on fixing this tomorrow, however it requires me to
modify the CdrServer, so we'll have to do some careful testing of
this with other link type definitions that use picklist
limitations. Then, to put it into production, we'll have to
promote the CdrServer to Bach, not just the new rules.

But, although it's more trouble than I had planned, the code
needs fixing. It only worked before in limited circumstances.

I'll work on it.

-----

Incidentally, while testing all of this, I didn't find any active
terms on Mahler that are both Index terms and Obsolete terms. I
presume that is expected, so I edited one of the terms (40134) to
make it obsolete for testing.

BZDATETIME::2010-09-17 17:59:29
BZCOMMENTOR::Alan Meyer
BZCOMMENT::4

I have modified the link property rule processing in the
CdrServer as described in the last comment and tested a number of
different document and link types for both link validation and
picklist generation.

I think everything is working correctly and in accordance with
our documentation of how link properties and picklist
manipulations work.

The modified server is installed on Mahler. I have also
installed a modified rule for Diagnosis links:

/Term/TermType/TermTypeName == "Index term" AND
/Term/TermType/TermTypeName -= "Obsolete term"

This says that a diagnosis term must be an index term. Obsolete
terms are okay but will not be included in any picklists for
diagnoses in XMetal.

If obsolete terms were not to be allowed, the rule would be:

/Term/TermType/TermTypeName == "Index term" AND
/Term/TermType/TermTypeName != "Obsolete term"

"-=" means allowed, but not to be included in a picklist.
"!=" means not allowed at all.

Right now the only active (not blocked) term on Mahler that is
both an Index term and an Obsolete term is CDR0000040134,

"monoclonal gammopathy of undetermined significance;Obsolete
term;Index term;Disease/diagnosis"

That was made obsolete by me. For QA, it may be desirable to
mark more terms as Obsolete.

This is ready for testing on Mahler.

BZDATETIME::2010-09-21 16:33:57
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::5

I have updated the title to reflect the new direction of this issue.

We have started testing this on Mahler.

BZDATETIME::2010-09-22 17:55:40
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::6

We have completed testing on Mahler. The solution worked well. Please promote to Franck so that we can QA and also see the impact on menu information, for example.

BZDATETIME::2010-09-30 23:24:57
BZCOMMENTOR::Alan Meyer
BZCOMMENT::7

I have backed up the old CdrServer on Franck and rebuilt a
new one with the required change. I also modified the Diagnosis
Link definition as required.

This is ready for user testing.

BZDATETIME::2010-10-06 17:52:32
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::8

(In reply to comment #7)
> I have backed up the old CdrServer on Franck and rebuilt a
> new one with the required change. I also modified the Diagnosis
> Link definition as required.
>
> This is ready for user testing.

I have been testing on Franck for a while and It appears not to be working very well.
For example, when I added a new TermTypeName of Obsolete to 42114 and 39756, I was still able to see them in the pick lists. The only time they don't show up on the pick list is when there is a SemanticType of Drug/Agent combination which is probably unrelated to this issue anyways.

BZDATETIME::2010-10-07 17:39:12
BZCOMMENTOR::Alan Meyer
BZCOMMENT::9

(In reply to comment #8)

> ...
> I have been testing on Franck for a while and It appears not to
> be working very well. For example, when I added a new
> TermTypeName of Obsolete to 42114 and 39756, I was still able
> to see them in the pick lists. The only time they don't show up
> on the pick list is when there is a SemanticType of Drug/Agent
> combination which is probably unrelated to this issue anyways.

After discussing this in the status meeting it appears that the
test conditions did not match the changes that were made.

The changes were ONLY made to the Diagnosis link type. The tests
above didn't involve Diagnosis links and so didn't invoke the
link validation modifications.

Here is what I think we need to do for testing:

To test what has been done so far:

Pick a Term document that has an appropriate SemanticType
value for assignment to a Diagnosis element.

Add a TermTypeName = "Obsolete term" to that Term.

Call up a protocol or other document in XMetal that uses
the Diagnosis link type.

Move to a Diagnosis element or create a new one.

Enter a leading substring that matches the leading
substring of the Term that got the new TermTypeName =
"Obsolete term".

Click Control-Enter to bring up a picklist.

If the software works correctly, the obsolete term should not
appear on the picklist. However, if the obsolete term is in
the document, it should not generate a validation error.

The main problem William encountered testing this was that he
didn't realize that only the Diagnosis link type was modified.
Someone (William? Margaret? Robin?) needs to review the link
types to see which other link types also need to filter out
obsolete terms from picklists.

Currently, we have ten link types that have Term documents as a
target. They are:

Condition
Diagnosis
Drug Term Link
Eligibility Criteria
Genetics Cancer Syndrome
SemanticType
Summary Drug
Summary Intervention
Term
TermRelations

Maybe all of them need the "Obsolete term" picklist restriction,
or maybe only some of them do.

To examine the link types in the Admin System click

Developers/System Administrators
Manage Linking Tables

They're all there in alphabetical order. Be sure not to change
anything in the link types. That's something we probably don't
want to do yet.

BZDATETIME::2010-10-12 10:46:17
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::10

We need the rule applied to three link types:
Diagnosis
Condition and
Genetics cancer syndrome

I went ahead and added:
/Term/TermType/TermTypeName -= "Obsolete term"
to condition and genetics cancer syndrome (since you already added it to diagnosis already) in MAHLER and they all worked without a problem when retrieving them from their specified link type elements.

However, I was still able to retrieve obsolete terms in the InterventionNameLink and the InterventionType links in pick lists of the protocol document. I was also able to retrieve obsolete terms from exclusion criteria link elements as well. So I am guessing that the rules for these link type elements may need to be modified as well. Generally, users should not be linking Diagnosis terms in the intervention links, for example. But my first choice would be to programmatically prevent them from doing that.

As a test, when I added the same rule (“/Term/TermType/TermTypeName -= "Obsolete term") to the Term link type (in MAHLER), the problem appeared to have gone away as I was no longer able to retrieve obsolete terms from the Intervention pick lists.

BZDATETIME::2010-10-12 16:36:29
BZCOMMENTOR::Alan Meyer
BZCOMMENT::11

(In reply to comment #10)
...
> As a test, when I added the same rule
> (/Term/TermType/TermTypeName -= "Obsolete term") to the Term
> link type (in MAHLER), the problem appeared to have gone away
> as I was no longer able to retrieve obsolete terms from the
> Intervention pick lists.

That is as it should be.

The definition of a link type is given in the combination of
"Can link from:" document type + element name
" Can link to:" document type

We haven't defined a link type called "InterventionType" or
"InterventionNameLink". We control those just from the "Term"
type. The "Term" link type encompasses a large number of links,
including the Intervention elements, but also links from these
other elements:

"FamilialCancerSyndrome"
"Gene"
"Term"
"Topic"
"GeneratedFrom"
"TermSetMember"

So, what we need to do depends on how we wish to treat those
other elements.

If "InterventionType" and "InterventionNameLink" are the ONLY
elements in the list that should not have obsolete terms in their
picklists, then we need to define a new link type, let's call it
"Intervention". The intervention related elements would need to
be removed from the Term link type and added to the new
Intervention link type. Then the picklist filtering rule should
be applied to that Intervention link type and removed from the
Term link type.

If, on the other hand, ALL of the above elements should have the
picklist filtering rule, then we can leave the link types alone
and just apply the rule to the Term link type, as you have done
on Mahler.

Let's discuss it at the meeting on Thursday. I can enter the
appropriate changes into the system after everyone has had a
chance to make the best decision.

BZDATETIME::2010-10-19 16:40:00
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::12

(In reply to comment #11)
> If, on the other hand, ALL of the above elements should have the
> picklist filtering rule, then we can leave the link types alone
> and just apply the rule to the Term link type, as you have done
> on Mahler.
>
> Let's discuss it at the meeting on Thursday. I can enter the
> appropriate changes into the system after everyone has had a
> chance to make the best decision.

I am not sure if this was discussed in last week's CDR meeting but I wanted to say that our 1st choice is to prevent obsolete terms from showing up in any pick list at all.

BZDATETIME::2010-10-21 21:52:22
BZCOMMENTOR::Alan Meyer
BZCOMMENT::13

I modified the link types on Franck and installed an updated
CdrServer there.

I did not modify two of the link types that already forbid
obsolete terms:

Summary Drug
Summary Intervention

Those types already specified:

AND /Term/TermType/TermTypeName != "Obsolete term"
^^

So for those two types Obsolete terms are completely invalid.
They don't need to be filtered out of picklists because only
valid terms will appear in the picklist.

We can change that if we need to to use:

AND /Term/TermType/TermTypeName -= "Obsolete term"
^^
For testing, I suggest going through the link types mentioned in
comment #9 and checking each. In some cases I added or removed
parentheses from the definitions. If I screwed something up an
Exception error will come back from the server.

BZDATETIME::2010-10-25 17:37:36
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::14

I completed testing this on Franck. I did not see any problems that should stop us from promoting this to Bach. Please promote to Bach.

BZDATETIME::2010-10-25 22:41:17
BZCOMMENTOR::Alan Meyer
BZCOMMENT::15

(In reply to comment #14)
> I completed testing this on Franck. I did not see any problems that should stop
> us from promoting this to Bach. Please promote to Bach.

Since this requires a restart of the CdrServer, I plan to do it tonight, while users are off the system.

I'll do it at 11 pm tonight. If anyone is using the system now and needs me to leave the server alone, please email me ASAP.

BZDATETIME::2010-10-25 23:21:52
BZCOMMENTOR::Alan Meyer
BZCOMMENT::16

I've replaced the CdrServer on Bach.

I'll install all of the new link rules tomorrow.

BZDATETIME::2010-10-26 23:07:36
BZCOMMENTOR::Alan Meyer
BZCOMMENT::17

I have applied all of the link rules that were on Franck to Bach.

I'm marking the issue resolved-fixed.

BZDATETIME::2010-11-01 16:43:20
BZCOMMENTOR::William Osei-Poku
BZCOMMENT::18

(In reply to comment #17)
> I have applied all of the link rules that were on Franck to Bach.
>
> I'm marking the issue resolved-fixed.

Verified on Bach. Issue closed. Thank you!